FLOSS Project Planets

Jean-Baptiste Onofré: MDC logging with Apache Karaf and Camel

Planet Apache - Sun, 2014-08-31 02:03

MDC (Mapped Diagnostic Context) logging is an interesting feature to log contextual messages.

It’s classic to want to log contextual messages in your application. For instance, we want to log the actions performed by an user (identified by an username or user id). As you have a lot of simultaneous users on your application, it’s easier to “follow” the log.

MDC is supported by several logging frameworks, like log4j or slf4j, and so by Karaf (thanks to pax-logging) as well.
The approach is pretty simple:

  1. You define the context using a key ID and a value for the key: MDC.put("userid", "user1");
  2. You use the logger as usual, the log messages to this logger will be contextual to the context: logger.debug("my message");
  3. After that, we can change the context by overriding the key: MDC.put("userid", "user2"); logger.debug("another message");

    Or you can remove the key, so to remove the context, and the log will be “global” (not local to a context):

    MDC.remove("userid"); // or MDC.clear() to remove all logger.debug("my global message");
  4. In the configuration, we can use pattern with %X{key} to log context. A pattern like %X{userid} - %m%n will result to a log file looking like: user1 - my message user2 - another message

In this blog, we will see how to use MDC in different cases (directly in your bundle, generic Karaf OSGi, and in Camel routes.

The source code of the blog post are available on my github: http://github.com/jbonofre/blog-mdc.

Using MDC in your application/bundle

The purpose here is to use slf4j MDC in our bundle and configure Karaf to create one log file per context.

To illustrate this, we will create multiple threads in the bundle, given a different context key for each thread:

package net.nanthrax.blog.mdc; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.slf4j.MDC; public class MdcExampleBean { private Logger logger = LoggerFactory.getLogger(MdcExampleBean.class); public void init() throws Exception { CycleThread thread1 = new CycleThread("thread1"); CycleThread thread2 = new CycleThread("thread2"); CycleThread thread3 = new CycleThread("thread3"); thread1.start(); thread2.start(); thread3.start(); } class CycleThread extends Thread { private String mdcContext; public CycleThread(String mdcContext) { this.mdcContext = mdcContext; } public void run() { MDC.put("threadId", mdcContext); for (int i = 0; i < 20; i++) { logger.info("Cycle {}", i); } } } }

After deploying this bundle in Karaf 3.0.1, we can see the log messages:

karaf@root()> bundle:install mvn:net.nanthrax.blog/mdc-bundle/1.0-SNAPSHOT karaf@root()> log:display ... 2014-08-30 09:44:25,594 | INFO | Thread-15 | MdcExampleBean | 78 - net.nanthrax.blog.mdc-bundle - 1.0.0.SNAPSHOT | Cycle 17 2014-08-30 09:44:25,594 | INFO | Thread-13 | MdcExampleBean | 78 - net.nanthrax.blog.mdc-bundle - 1.0.0.SNAPSHOT | Cycle 19 2014-08-30 09:44:25,594 | INFO | Thread-15 | MdcExampleBean | 78 - net.nanthrax.blog.mdc-bundle - 1.0.0.SNAPSHOT | Cycle 18 2014-08-30 09:44:25,595 | INFO | Thread-15 | MdcExampleBean | 78 - net.nanthrax.blog.mdc-bundle - 1.0.0.SNAPSHOT | Cycle 19

Now, we can setup the Karaf etc/org.ops4j.pax.logging.cfg file to use our MDC. For that, we add a MDCSiftingAppender, providing the threadId as MDC key, and displaying the threadId in the log message pattern. We will create one log file per key (threadId in our case), and finally, we add this appender to the rootLogger:

... log4j.rootLogger=INFO, out, mdc-bundle, osgi:* ... # MDC Bundle appender log4j.appender.mdc-bundle=org.apache.log4j.sift.MDCSiftingAppender log4j.appender.mdc-bundle.key=threadId log4j.appender.mdc-bundle.default=unknown log4j.appender.mdc-bundle.appender=org.apache.log4j.FileAppender log4j.appender.mdc-bundle.appender.layout=org.apache.log4j.PatternLayout log4j.appender.mdc-bundle.appender.layout.ConversionPattern=%d | %-5.5p | %X{threadId} | %m%n log4j.appender.mdc-bundle.appender.file=${karaf.data}/log/mdc-bundle-$\\{threadId\\}.log log4j.appender.mdc-bundle.appender.append=true ...

Now, in the Karaf data/log folder, we can see:

mdc-bundle-thread1.log mdc-bundle-thread2.log mdc-bundle-thread3.log

each file containing the log messages contextual to the thread:

$ cat data/log/mdc-bundle-thread1.log 2014-08-30 09:54:48,287 | INFO | thread1 | Cycle 0 2014-08-30 09:54:48,298 | INFO | thread1 | Cycle 1 2014-08-30 09:54:48,298 | INFO | thread1 | Cycle 2 2014-08-30 09:54:48,299 | INFO | thread1 | Cycle 3 2014-08-30 09:54:48,299 | INFO | thread1 | Cycle 4 ... $ cat data/log/mdc-bundle-thread2.log 2014-08-30 09:54:48,287 | INFO | thread2 | Cycle 0 2014-08-30 09:54:48,298 | INFO | thread2 | Cycle 1 2014-08-30 09:54:48,298 | INFO | thread2 | Cycle 2 2014-08-30 09:54:48,299 | INFO | thread2 | Cycle 3 2014-08-30 09:54:48,299 | INFO | thread2 | Cycle 4 2014-08-30 09:54:48,299 | INFO | thread2 | Cycle 5 ...

In addition, Karaf “natively” provides OSGi MDC data that we can use.

Using Karaf OSGi MDC

So, in Karaf, you can use directly some OSGi headers for MDC logging, especially the bundle name.

We can use this MDC key to create one log file per bundle.

Karaf already provides a pre-defined appender configuration in etc/org.ops4j.pax.logging.cfg:

... # Sift appender log4j.appender.sift=org.apache.log4j.sift.MDCSiftingAppender log4j.appender.sift.key=bundle.name log4j.appender.sift.default=karaf log4j.appender.sift.appender=org.apache.log4j.FileAppender log4j.appender.sift.appender.layout=org.apache.log4j.PatternLayout log4j.appender.sift.appender.layout.ConversionPattern=%d{ISO8601} | %-5.5p | %-16.16t | %-32.32c{1} | %m%n log4j.appender.sift.appender.file=${karaf.data}/log/$\\{bundle.name\\}.log log4j.appender.sift.appender.append=true ...

The only thing that we have to do is to add this appender to the rootLogger:

log4j.rootLogger=INFO, out, sift, osgi:*

Now, in the Karaf data/log folder, we can see one file per bundle:

data/log$ ls -1 karaf.log net.nanthrax.blog.mdc-bundle.log org.apache.aries.blueprint.core.log org.apache.aries.jmx.core.log org.apache.felix.fileinstall.log org.apache.karaf.features.core.log org.apache.karaf.region.persist.log org.apache.karaf.shell.console.log org.apache.sshd.core.log

Especially, we can see our mdc-bundle, containing the log messages “local” to the bundle.

However, if this approach works great, it doesn’t always create interesting log file. For instance, when you use Camel, using OSGi headers for MDC logging will gather most of the log messages into the camel-core bundle log file, so, not really contextual to something or easy to read/seek.

The good news is that Camel also provides MDC logging support.

Using Camel MDC

If Camel provides MDC logging support, it’s not enabled by default. It’s up to you to enable it on the camel context.

Once enabled, Camel provides the following MDC logging properties:

  • camel.exchangeId providing the exchange ID
  • camel.messageId providing the message ID
  • camel.routeId providing the route ID
  • camel.contextId providing the Camel Context ID
  • camel.breadcrumbId providing an unique id used for tracking messages across transports
  • camel.correlationId providing the correlation ID of the exchange (if it’s correlated, for instance like in Splitter EIP)
  • camel.trasactionKey providing the ID of the transaction (for transacted exchange).

To enable the MDC logging, you have to:

  • if you use the Blueprint or Spring XML DSL: <camelContext xmlns="http://camel.apache.org/schema/blueprint" useMDCLogging="true">
  • if you use the Java DSL: CamelContext context = ... context.setUseMDCLogging(true);
  • using the Talend ESB studio, you have to use a cConfig component from the palette:

So, let say, we create the following route using the Blueprint DSL:

<?xml version="1.0" encoding="UTF-8"?> <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"> <camelContext xmlns="http://camel.apache.org/schema/blueprint" useMDCLogging="true"> <route id="my-route"> <from uri="timer:fire?period=5000"/> <setBody> <constant>Hello Blog</constant> </setBody> <to uri="log:net.nanthrax.blog?level=INFO"/> </route> </camelContext> </blueprint>

We want to create one log file per route (using the routeId). So, we update the Karaf etc/org.ops4j.pax.logging.cfg file to add a MDC sifting appender using the Camel MDC properties, and we add this appender to the rootLogger:

... log4j.rootLogger=INFO, out, camel-mdc, osgi:* ... # Camel MDC appender log4j.appender.camel-mdc=org.apache.log4j.sift.MDCSiftingAppender log4j.appender.camel-mdc.key=camel.routeId log4j.appender.camel-mdc.default=unknown log4j.appender.camel-mdc.appender=org.apache.log4j.FileAppender log4j.appender.camel-mdc.appender.layout=org.apache.log4j.PatternLayout log4j.appender.camel-mdc.appender.layout.ConversionPattern=%d{ISO8601} | %-5.5p | %-16.16t | %-32.32c{1} | %X{camel.exchangeId} | %m%n log4j.appender.camel-mdc.appender.file=${karaf.data}/log/camel-$\\{camel.routeId\\}.log log4j.appender.camel-mdc.appender.append=true ...

The camel-mdc appender will create one log file by route (named camel-(routeId).log). The log messages will contain the exchange ID.

We start Karaf, and after the installation of the camel-blueprint feature, we can drop our route.xml directly in the deploy folder:

karaf@root()> feature:repo-add camel 2.12.1 Adding feature url mvn:org.apache.camel.karaf/apache-camel/2.12.1/xml/features karaf@root()> feature:install camel-blueprint cp route.xml apache-karaf-3.0.1/deploy/

Using log:display command in Karaf, we can see the messages for our route:

karaf@root()> log:display

2014-08-31 08:58:24,176 | INFO | 0 – timer://fire | blog | 85 – org.apache.camel.camel-core – 2.12.1 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: Hello Blog]
2014-08-31 08:58:29,176 | INFO | 0 – timer://fire | blog | 85 – org.apache.camel.camel-core – 2.12.1 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: Hello Blog]

Now, if we go into the Karaf data/log folder, we can see the log file for our route:

$ ls -1 data/log camel-my-route.log ...

If we take a look in the camel-my-route.log file, we can see the messages contextual to the route, including the exchange ID:

2014-08-31 08:58:19,196 | INFO | 0 - timer://fire | blog | ID-latitude-57336-1409468297774-0-2 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: Hello Blog] 2014-08-31 08:58:24,176 | INFO | 0 - timer://fire | blog | ID-latitude-57336-1409468297774-0-4 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: Hello Blog] 2014-08-31 08:58:29,176 | INFO | 0 - timer://fire | blog | ID-latitude-57336-1409468297774-0-6 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: Hello Blog] 2014-08-31 08:58:34,176 | INFO | 0 - timer://fire | blog | ID-latitude-57336-1409468297774-0-8 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: Hello Blog]
Categories: FLOSS Project Planets

Mukul Gandhi: about xml schema 1.1 book

Planet Apache - Sun, 2014-08-31 00:48
A new post perhaps with the right reason. I've written a book with title "W3C XML Schema 1.1 for Beginners", that essentially covers 1.1 version of XSD language, but also covers 1.0 version of XSD language in fair detail. Noah Mendelsohn has kindly written forward text of this book.

Here are the links to this book on web,

http://www.shroffpublishers.com/detail.aspx?cat=42&title=6176 , publisher's site where a TOC is available
http://www.amazon.in/W3c-XML-Schema-1-1-Beginners/dp/1619030187
http://www.amazon.com/W3C-XML-Schema-1-1-Beginners/dp/1619030187

Just thought of making this aware to a larger community.
Categories: FLOSS Project Planets

Graeme Cross: Notes from MPUG, August 2014

Planet Python - Sat, 2014-08-30 23:55

A strong line-up of speakers and the post-PyCon AU buzz meant that we had about 35 people at the August Melbourne Python Users Group meeting. There were five presentations for the night and here are my random notes.

1. Highlights from PyCon AU 2014

Various people contributed their highlights of PyCon AU 2014, which was in Brisbane in early August (and the reason why the MPUG meeting was moved back a week). The conference videos are now up on the Youtube channel.

2. Javier Candeira: “Don’t monkeypatch None!”

Javier was inspired by discussion at PyCon AU about PEP 336 (Make None Callable) and decided to implement it, with some inspiration from previous PyCon AU talks by Ryan Kelly, Richard Jones (“Don’t do this!”), Nick Coghlan (“Elegant and ugly hacks in CPython”) and Larry Hastings.

He has implemented quiet_None, an analog to quiet_NaN. He walked us through the prototype code, with a deviation into Haskell’s Maybe monad along the way.

A few random notes:

  • Uses the ContextDecorator class
  • Module: “forbiddenfruit” and “curse” function
  • Won’t work on stackless or IronPython (as they have no working implementation of sys._getframe()
  • Problems with exceptions, so the code needs to monkeypatch Exception!
3. Juan Nunez-Iglesias: SciPy 2014, a summary

Juan works at VLSCI and is a scikit-image contributor

  • Great quote: Greg Wilson (keynote) on Software Carpentry: “we save researchers a day a week for the rest of their careers”
  • Lots of core devs from biggest names in Python scientific stack
  • Excellent tutorials; massive geoscience track, massive education track
  • EuroSciPy seems to be more academic
  • PyCon AU – very web development focus

SciPy 2014 highlights:

  • Michael Droettboom – Airspeed Velocity: code benchmarking tool with web interface and plots
  • Paul Ivanov – IPython vimception: vim bindings for IPython notebook
  • Damian Avila – RISE: live presentations from IPython notebooks
  • MinRK (from core IPython team) – IPython Parallel tutorial
  • Brian Granger & ????
  • Aaron Meurer: Conda – good package manager for C extensions as well as Python packages
  • Matt Rocklin, Mark Wiebe: Blaze – abstracting away tabular data

Recommended lightning talks:

  • Project Jupyter
  • Waffles

On Juan’s “to watch” list:

  • Chris Fonnesbeck PyMC tutorial (see Jake Vanderplas’s Bayesian vs frequentist posts)
  • David Sanders: Julia tutorial
  • Reproducible science tutorial (lots of buzz on this tutorial)
  • Geoscience and education tracks
  • Diversity: 1.5% women in 2013, 15% in 2014 – indicative of a big problem in the industry
4. Rory Hart: Microservices in Python

This was a repeat of Rory’s PyCon AU 2014 talk (here’s the video).

5. Nick Farrell: sux

Nick gave a repeat of his PyCon AU 2014 lightning talk on the sux library, which allows you to transparently use Python 2 packages in Python 3.

Categories: FLOSS Project Planets

Alexander Wirt: cgit on alioth.debian.org

Planet Debian - Sat, 2014-08-30 23:52

Recently I was doing some work on the alioth infrastructure like fixing things or cleaning up things.

One of the more visible things I done was the switch from gitweb to cgit. cgit is a lot of faster and looks better than gitweb.

The list of repositories is generated every hour. The move also has the nice effect that user repositories are available via the cgit index again.

I don’t plan to disable the old gitweb, but I created a bunch of redirect rules that - hopefully - redirect most use cases of gitweb to the equivalent cgit url.

If I broke something, please tell me, if I missed a common use case, please tell me. You can usually reach me on #alioth@oftc or via mail (formorer@d.o)

People also asked me to upload my cgit package to Debian, the package is now waiting in NEW. Thanks to Nicolas Dandrimont (olasd) we also have a patch included that generates proper HTTP returncodes if repos doesn’t exist.

Categories: FLOSS Project Planets

Dave Behnke: Fun with SQLAlchemy

Planet Python - Sat, 2014-08-30 19:30

This is a little experiment I created with SQLAlchemy. In this notebook, I'm using sqlite to create a table, and doing some operations such as deleting all the rows in the table and inserting a list of items.

In [2]: # connection is a connection to the database from a pool of connections connection = engine.connect() # meta will be used to reflect the table later meta = MetaData() 2014-08-10 21:10:16,410 INFO sqlalchemy.engine.base.Engine SELECT CAST(&apostest plain returns&apos AS VARCHAR(60)) AS anon_1 INFO:sqlalchemy.engine.base.Engine:SELECT CAST(&apostest plain returns&apos AS VARCHAR(60)) AS anon_1 2014-08-10 21:10:16,411 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() 2014-08-10 21:10:16,413 INFO sqlalchemy.engine.base.Engine SELECT CAST(&apostest unicode returns&apos AS VARCHAR(60)) AS anon_1 INFO:sqlalchemy.engine.base.Engine:SELECT CAST(&apostest unicode returns&apos AS VARCHAR(60)) AS anon_1 2014-08-10 21:10:16,414 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() In [3]: # create the table if it doesn't exist already connection.execute("create table if not exists test ( id integer primary key autoincrement, name text )") 2014-08-10 21:10:17,212 INFO sqlalchemy.engine.base.Engine create table if not exists test ( id integer primary key autoincrement, name text ) INFO:sqlalchemy.engine.base.Engine:create table if not exists test ( id integer primary key autoincrement, name text ) 2014-08-10 21:10:17,214 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() 2014-08-10 21:10:17,217 INFO sqlalchemy.engine.base.Engine COMMIT INFO:sqlalchemy.engine.base.Engine:COMMIT Out[3]: <sqlalchemy.engine.result.ResultProxy at 0x105083e48> In [4]: #reflects all the tables in the current connection meta.reflect(bind=engine) 2014-08-10 21:10:17,986 INFO sqlalchemy.engine.base.Engine SELECT name FROM (SELECT * FROM sqlite_master UNION ALL SELECT * FROM sqlite_temp_master) WHERE type=&apostable&apos ORDER BY name INFO:sqlalchemy.engine.base.Engine:SELECT name FROM (SELECT * FROM sqlite_master UNION ALL SELECT * FROM sqlite_temp_master) WHERE type=&apostable&apos ORDER BY name 2014-08-10 21:10:17,987 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() 2014-08-10 21:10:17,989 INFO sqlalchemy.engine.base.Engine PRAGMA table_info("sqlite_sequence") INFO:sqlalchemy.engine.base.Engine:PRAGMA table_info("sqlite_sequence") 2014-08-10 21:10:17,990 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() 2014-08-10 21:10:17,993 INFO sqlalchemy.engine.base.Engine PRAGMA foreign_key_list("sqlite_sequence") INFO:sqlalchemy.engine.base.Engine:PRAGMA foreign_key_list("sqlite_sequence") 2014-08-10 21:10:17,995 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() 2014-08-10 21:10:17,997 INFO sqlalchemy.engine.base.Engine PRAGMA index_list("sqlite_sequence") INFO:sqlalchemy.engine.base.Engine:PRAGMA index_list("sqlite_sequence") 2014-08-10 21:10:17,997 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() 2014-08-10 21:10:17,999 INFO sqlalchemy.engine.base.Engine PRAGMA table_info("test") INFO:sqlalchemy.engine.base.Engine:PRAGMA table_info("test") 2014-08-10 21:10:18,000 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() 2014-08-10 21:10:18,001 INFO sqlalchemy.engine.base.Engine PRAGMA foreign_key_list("test") INFO:sqlalchemy.engine.base.Engine:PRAGMA foreign_key_list("test") 2014-08-10 21:10:18,002 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() 2014-08-10 21:10:18,004 INFO sqlalchemy.engine.base.Engine PRAGMA index_list("test") INFO:sqlalchemy.engine.base.Engine:PRAGMA index_list("test") 2014-08-10 21:10:18,005 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() In [5]: # grabs a Table object from meta test = meta.tables['test'] test Out[5]: Table(&apostest&apos, MetaData(bind=None), Column(&aposid&apos, INTEGER(), table=<test>, primary_key=True, nullable=False), Column(&aposname&apos, TEXT(), table=<test>), schema=None) In [6]: # cleans out all the rows in the test table result = connection.execute(test.delete()) print("Deleted %d row(s)" % result.rowcount) 2014-08-10 21:10:22,659 INFO sqlalchemy.engine.base.Engine DELETE FROM test INFO:sqlalchemy.engine.base.Engine:DELETE FROM test 2014-08-10 21:10:22,661 INFO sqlalchemy.engine.base.Engine () INFO:sqlalchemy.engine.base.Engine:() 2014-08-10 21:10:22,662 INFO sqlalchemy.engine.base.Engine COMMIT INFO:sqlalchemy.engine.base.Engine:COMMIT Deleted 11 row(s) In [7]: # create a list of names to be inserting into the test table names = ['alpha', 'bravo', 'charlie', 'delta', 'epsilon', 'foxtrot', 'golf', 'hotel', 'india', 'juliet', 'lima'] In [8]: # perform multiple inserts, the list is converted on the fly into a dictionary with the name field. result = connection.execute(test.insert(), [{'name': name} for name in names]) print("Inserted %d row(s)" % result.rowcount) 2014-08-10 21:10:26,580 INFO sqlalchemy.engine.base.Engine INSERT INTO test (name) VALUES (?) INFO:sqlalchemy.engine.base.Engine:INSERT INTO test (name) VALUES (?) 2014-08-10 21:10:26,582 INFO sqlalchemy.engine.base.Engine ((&aposalpha&apos,), (&aposbravo&apos,), (&aposcharlie&apos,), (&aposdelta&apos,), (&aposepsilon&apos,), (&aposfoxtrot&apos,), (&aposgolf&apos,), (&aposhotel&apos,) ... displaying 10 of 11 total bound parameter sets ... (&aposjuliet&apos,), (&aposlima&apos,)) INFO:sqlalchemy.engine.base.Engine:((&aposalpha&apos,), (&aposbravo&apos,), (&aposcharlie&apos,), (&aposdelta&apos,), (&aposepsilon&apos,), (&aposfoxtrot&apos,), (&aposgolf&apos,), (&aposhotel&apos,) ... displaying 10 of 11 total bound parameter sets ... (&aposjuliet&apos,), (&aposlima&apos,)) 2014-08-10 21:10:26,583 INFO sqlalchemy.engine.base.Engine COMMIT INFO:sqlalchemy.engine.base.Engine:COMMIT Inserted 11 row(s) In [9]: # query the rows with select, the where clause is included for demostration # it can be omitted result = connection.execute(select([test]).where(test.c.id > 0)) 2014-08-10 21:10:28,528 INFO sqlalchemy.engine.base.Engine SELECT test.id, test.name FROM test WHERE test.id > ? INFO:sqlalchemy.engine.base.Engine:SELECT test.id, test.name FROM test WHERE test.id > ? 2014-08-10 21:10:28,529 INFO sqlalchemy.engine.base.Engine (0,) INFO:sqlalchemy.engine.base.Engine:(0,) In [10]: # show the results for row in result: print("id=%d, name=%s" % (row['id'], row['name'])) id=56, name=alpha id=57, name=bravo id=58, name=charlie id=59, name=delta id=60, name=epsilon id=61, name=foxtrot id=62, name=golf id=63, name=hotel id=64, name=india id=65, name=juliet id=66, name=lima In []:

Categories: FLOSS Project Planets

Francois Marier: Outsourcing your webapp maintenance to Debian

Planet Debian - Sat, 2014-08-30 16:45

Modern web applications are much more complicated than the simple Perl CGI scripts or PHP pages of the past. They usually start with a framework and include lots of external components both on the front-end and on the back-end.

Here's an example from the Node.js back-end of a real application:

$ npm list | wc -l 256

What if one of these 256 external components has a security vulnerability? How would you know and what would you do if of your direct dependencies had a hard-coded dependency on the vulnerable version? It's a real problem and of course one way to avoid this is to write everything yourself. But that's neither realistic nor desirable.

However, it's not a new problem. It was solved years ago by Linux distributions for C and C++ applications. For some reason though, this learning has not propagated to the web where the standard approach seems to be to "statically link everything".

What if we could build on the work done by Debian maintainers and the security team?

Case study - the Libravatar project

As a way of discussing a different approach to the problem of dependency management in web applications, let me describe the decisions made by the Libravatar project.

Description

Libravatar is a federated and free software alternative to the Gravatar profile photo hosting site.

From a developer point of view, it's a fairly simple stack:

The service is split between the master node, where you create an account and upload your avatar, and a few mirrors, which serve the photos to third-party sites.

Like with Gravatar, sites wanting to display images don't have to worry about a complicated protocol. In a nutshell, all that a site needs to do is hash the user's email and add that hash to a base URL. Where the federation kicks in is that every email domain is able to specify a different base URL via an SRV record in DNS.

For example, francois@debian.org hashes to 7cc352a2907216992f0f16d2af50b070 and so the full URL is:

http://cdn.libravatar.org/avatar/7cc352a2907216992f0f16d2af50b070

whereas francois@fmarier.org hashes to 0110e86fdb31486c22dd381326d99de9 and the full URL is:

http://fmarier.org/avatar/0110e86fdb31486c22dd381326d99de9

due to the presence of an SRV record on fmarier.org.

Ground rules

The main rules that the project follows is to:

  1. only use Python libraries that are in Debian
  2. use the versions present in the latest stable release (including backports)
Deployment using packages

In addition to these rules around dependencies, we decided to treat the application as if it were going to be uploaded to Debian:

  • It includes an "upstream" Makefile which minifies CSS and JavaScript, gzips them, and compiles PO files (i.e. a "build" step).
  • The Makefile includes a test target which runs the unit tests and some lint checks (pylint, pyflakes and pep8).
  • Debian packages are produced to encode the dependencies in the standard way as well as to run various setup commands in maintainer scripts and install cron jobs.
  • The project runs its own package repository using reprepro to easily distribute these custom packages.
  • In order to update the repository and the packages installed on servers that we control, we use fabric, which is basically a fancy way to run commands over ssh.
  • Mirrors can simply add our repository to their apt sources.list and upgrade Libravatar packages at the same time as their system packages.
Results

Overall, this approach has been quite successful and Libravatar has been a very low-maintenance service to run.

The ground rules have however limited our choice of libraries. For example, to talk to our queuing system, we had to use the raw Python bindings to the C Gearman library instead of being able to use a nice pythonic library which wasn't in Debian squeeze at the time.

There is of course always the possibility of packaging a missing library for Debian and maintaining a backport of it until the next Debian release. This wouldn't be a lot of work considering the fact that responsible bundling of a library would normally force you to follow its releases closely and keep any dependencies up to date, so you may as well share the result of that effort. But in the end, it turns out that there is a lot of Python stuff already in Debian and we haven't had to package anything new yet.

Another thing that was somewhat scary, due to the number of packages that were going to get bumped to a new major version, was the upgrade from squeeze to wheezy. It turned out however that it was surprisingly easy to upgrade to wheezy's version of Django, Apache and Postgres. It may be a problem next time, but all that means is that you have to set a day aside every 2 years to bring everything up to date.

Problems

The main problem we ran into is that we optimized for sysadmins and unfortunately made it harder for new developers to setup their environment. That's not very good from the point of view of welcoming new contributors as there is quite a bit of friction in preparing and testing your first patch. That's why we're looking at encoding our setup instructions into a Vagrant script so that new contributors can get started quickly.

Another problem we faced is that because we use the Debian version of jQuery and minify our own JavaScript files in the build step of the Makefile, we were affected by the removal from that package of the minified version of jQuery. In our setup, there is no way to minify JavaScript files that are provided by other packages and so the only way to fix this would be to fork the package in our repository or (preferably) to work with the Debian maintainer and get it fixed globally in Debian.

One thing worth noting is that while the Django project is very good at issuing backwards-compatible fixes for security issues, sometimes there is no way around disabling broken features. In practice, this means that we cannot run unattended-upgrades on our main server in case something breaks. Instead, we make use of apticron to automatically receive email reminders for any outstanding package updates.

On that topic, it can occasionally take a while for security updates to be released in Debian, but this usually falls into one of two cases:

  1. You either notice because you're already tracking releases pretty well and therefore could help Debian with backporting of fixes and/or testing;
  2. or you don't notice because it has slipped through the cracks or there simply are too many potential things to keep track of, in which case the fact that it eventually gets fixed without your intervention is a huge improvement.

Finally, relying too much on Debian packaging does prevent Fedora users (a project that also makes use of Libravatar) from easily contributing mirrors. Though if we had a concrete offer, we would certainly look into creating the appropriate RPMs.

Is it realistic?

It turns out that I'm not the only one who thought about this approach, which has been named "debops". The same day that my talk was announced on the DebConf website, someone emailed me saying that he had instituted the exact same rules at his company, which operates a large Django-based web application in the US and Russia. It was pretty impressive to read about a real business coming to the same conclusions and using the same approach (i.e. system libraries, deployment packages) as Libravatar.

Regardless of this though, I think there is a class of applications that are particularly well-suited for the approach we've just described. If a web application is not your full-time job and you want to minimize the amount of work required to keep it running, then it's a good investment to restrict your options and leverage the work of the Debian community to simplify your maintenance burden.

The second criterion I would look at is framework maturity. Given the 2-3 year release cycle of stable distributions, this approach is more likely to work with a mature framework like Django. After all, you probably wouldn't compile Apache from source, but until recently building Node.js from source was the preferred option as it was changing so quickly.

While it goes against conventional wisdom, relying on system libraries is a sustainable approach you should at least consider in your next project. After all, there is a real cost in bundling and keeping up with external dependencies.

This blog post is based on a talk I gave at DebConf 14: slides, video.

Categories: FLOSS Project Planets

Dave Behnke: Back to Python

Planet Python - Sat, 2014-08-30 15:38

After some "soul searching" and investigation between go and python over the last few months. I've decided to come back to Python.

My Experience

I spent a couple months researching and developing with Go. I even bought a pre-released book (Go in Action - http://www.manning.com/ketelsen/). The concurrency chapter wasn't written quite yet so I ended up looking elsewhere. I eventually found an excellent book explaining concurrency concepts of Go through my safari account (https://www.safaribooksonline.com/). The book is entitled Mastering Concurrency in Go. (https://www.packtpub.com/application-development/mastering-concurrency-go)

After going through a couple of programming exercises using Go, I started to think to myself, how would I do this in Python. It started to click in my brain that on a conceptional level a goroutine is similar to a async coroutine in Python. The main difference is that Go was designed from the beginning to be concurrent. Python it requires a little more work.

I'll make a long story short. Go is a good language, I will probably use it for specific problems to solve. I'm more familiar with Python. The passionate Python community makes me proud to be a part of it. I like having access to many interesting modules and packages. Ipython, Flask, Djano, Sqlalchemy just to name a few.

I look forward to continuing to work with Python and share code examples where I can. Stay tuned!

Categories: FLOSS Project Planets

Q&A with the board of KDE e.V. at Akademy

Planet KDE - Sat, 2014-08-30 15:13

Only a few days left until Akademy. I’m looking forward to meeting old friends again and making new ones. There are many exciting talks in the program that I want to see. Have a look!

The board is going to do a Q&A session on Saturday afternoon. We want to give more people a chance to ask questions than just the ones attending Akademy. I started a wiki page to collect them and we’ll try to answer as many as we can.

Categories: FLOSS Project Planets

Michael McCandless: Scoring tennis using finite-state automata

Planet Apache - Sat, 2014-08-30 14:51
For some reason having to do with the medieval French, the scoring system for tennis is very strange.

In actuality, the game is easy to explain: to win, you must score at least 4 points and win by at least 2. Yet in practice, you are supposed to use strange labels like "love" (0 points), "15" (1 point), "30" (2 points), "40" (3 points), "deuce" (3 or more points each, and the players are tied), "all" (players are tied) instead of simply tracking points as numbers, as other sports do.

This is of course wildly confusing to newcomers. Fortunately, the convoluted logic is easy to express as a finite-state automaton (FSA):



The game begins in the left-most (unlabeled) state, and then each time either player 1 (red) or player 2 (blue) scores, you advance to the corresponding state to know how to say the score properly in tennis-speak. In each state, player 1's score is first followed by player 2's; for example "40 30" means player 1 has scored 3 points and player 2 has scored 2 and "15 all" means both players have scored once. "adv 2" means player 2 is ahead by 1 point and will win if s/he scores again.

There are only 20 states, and there are cycles which means a tennis game can in fact go on indefinitely, if the players pass back and forth through the "deuce" (translation: game is tied) state.

This FSA is correct, and if you watch a Wimbledon match, for example, you'll see the game advance through precisely these states.

Minimization

Yet for an FSA, merely being correct is not good enough!

It should also strive to be minimal, and surprisingly this FSA is not: if you build this Automaton in Luceneand minimize it, you'll discover that there are some wasted states! This means 20 states is overkill when deciding who won the game.

Specifically, there is no difference between the "30 all" and "deuce" states, nor between the "30 40" and "adv 2" states, nor between the "40 30" and "adv 1" states. From either state in each of these pairs, there is no sequence of player 1 / player 2 scoring that will result in a different final outcome (this is in principle how the minimization process identifies indistinguishable states).

Therefore, there's no point in keeping those states, and you can safely use this smaller 17-state FSA (10% smaller!) to score your tennis games instead:



For example, from "15 30", if player 1 scores, you go straight to "deuce" and don't bother with the redundant "30 30" state.

Another (simpler?) way to understand why these states are wasted is to recognize that the finite state machine is tracking two different pieces of information: first, how many points ahead player 1 is (since a player must win by 2 points) and second, how many points have been scored (since a player must score at least 4 points to win).

Once enough points (4 or more) have been scored by either player, their absolute scores no longer matter. All that matters is the relative score: whether player 1 is ahead by 1, equal, or behind by 1. For example, we don't care if the score is 197 to 196 or 6 to 5: they are the same thing.

Yet, early on, the FSA must also track the absolute scores, to ensure at least 4 points were scored by the winner. With the original 20-state FSA, the crossover between these two phases was what would have been "40 40" (each player scored 3 points). But in the minimal machine, the crossover became "30 30" (each player scored 2 points), which is safe since each player must still "win by 2" so if player 1 scores 2 points from "30 30", that means player 1 scored 4 points overall.

FSA minimization saved only 3 states for the game of tennis, resulting in a 10% smaller automaton, and maybe this simplifies keeping track of scores in your games by a bit, but in other FSA applications in Lucene, such as the analyzing suggester, MemoryPostingsFormatand the terms index, minimization is vital since saves substantial disk and RAM for Lucene applications!
Categories: FLOSS Project Planets

Bryan Pendleton: Ireland Day One: Arriving and Driving

Planet Apache - Sat, 2014-08-30 14:47

Because of the carriers we used, we found ourselves flying in and out of the brand new Terminal 2 (the "Queen's Terminal") at London Heathrow. This is truly a beautiful facility and I was very impressed by it: spacious but easy to navigate, with nice shops and restaurants and plenty of seating to accomodate travelers with time to spare.

Perhaps I should say a bit more about those "shops and restaurants", though. We had two very nice meals at the WonderTree Cafe in Terminal 2, and I recommend it highly. But the signature shops and restaurants for which Terminal 2 is known are, well, pretty high end (Burberry, Gucci, Bulgari, Bottega Veneta, Harrods, etc.); perhaps you might want to have a look at the menu for the Prunier Seafood Bar?

Enough of that; it's on to Ireland!

Our Aer Lingus flight to Shannon airport takes barely an hour, although unfortunately the cloud cover is extensive and I am only able to see a little bit at takeoff and a little bit at landing, enough though I suspect we flew over some beautiful territory (Wales, St George's Channel, as well as most of south-central Ireland).

Shannon Airport is small and friendly and easy to handle. We simply walked across the street from the terminal to the car park to pick up our rented Renault and we were off on the Great Ireland Adventure!

We deliberately scheduled an extremely short trip on our first day in Ireland, because we knew that it would take some time to get the hang of driving in Ireland.

Really, though, there are only four elements to master in order to drive in Ireland:

  1. Drive on the left.

    In Ireland, you drive on the opposite side of the road as compared to America, and the steering wheel is on the opposite side of the vehicle. I kept chanting to myself: "look to the right, keep to the left". But really, this didn't take very long to get used to; I was rather surprised about that.

    It did seem like, each morning, I would have to re-remind myself about how things go, and it was a bit of a challenge to re-train myself to look over the correct shoulder when I was checking for things in my mirrors.

  2. Get the hang of the roundabouts.

    In Ireland (at least in the areas we visited), there are almost no traffic lights or stop signs. Instead, most intersections are roundabouts, so you need to know how to navigate a roundabout properly. Mostly that means:

    • Look to your right when entering the roundabout
    • Yield to the traffic already in the roundabout before entering it
    • Pay attention to the sign as you're approaching the roundabout, so that you know which exit you want as you leave the roundabout
    Heck, it's really not that hard: just look at the video.
  3. Distances and speeds are in Kilometers, not in Miles.

    So when you see that the speed limit through town is 50 km/hr, you instinctively might think that's rather startling, because 50 MPH in America is far too fast to drive through these tiny villages. But, of course, 50 km/hr is much slower, and perfectly appropriate.

  4. Drive country roads carefully.

    In the part of Ireland we were visiting, there were only a few super speedways (called "motorways" in Ireland). Most of the roads we found ourselves driving were absolutely gorgeous, among the most scenic roads I've ever driven in my life, but they were narrow, curvy, and had very restricted visibility. So just

    • Go slow
    • Give the other drivers plenty of room
    • Be patient
    • Enjoy the beauty of the roads
    • Oh, and yes: go slow!
    It frequently amused us to see a sign on a country road telling us that the speed limit was 80 km/hr or even 100 km/hr, when the road itself could clearly be driven only at 40, 50, or 60 km/hr.

    Driving the country roads safely was definitely the most challenging part of driving in Ireland, but I'm pleased to report that we had a trouble-free time of it!

So, anyway, we left the airport and got on the road.

A few miles from Shannon Airport is Bunratty Castle and Folk Park, an interesting place.

If you've read McCarthy's Bar, you'll recall that Bunratty Castle is the site of the funniest episode in the entire book, an absolutely hilarious adventure involving country roads, pubs, a medieval banquet, and two sisters from Philadelphia who have brought their children to Ireland.

Really, you have to read the book.

I wish we had taken the time to go in to Bunratty Castle and take the tour and visit the park, but we were still Ireland novices and I made a bad decision and decided to push on. This was perhaps one of only two bad decisions I made in Ireland about where to go and what to do, so I'm not going to beat myself up about it too much, but I'm definitely a bit disappointed to have only seen the castle from the outside (where it is still quite impressive).

So after waving to some of the children who were Way Up There at the top of the castle, we climb back into the car and venture on.

We drive into downtown Limerick and walk through the busy commercial center. We get some nice shots of King John's Castle from the river front, pick up a few necessities from the Dunnes Store in Limerick ("Umbrellas! How did we fail to pack umbrellas?"), and on we go.

We take the motorway out of Limerick, then exit to the N21. Just 4 km before Adare, traffic crawls to a stop and Garda with alert lights flashing zip by us. We crawl into Adare at barely 10 km/hr, but the scenery is delightful and, did I mention this, WE'RE IN IRELAND!

As we reach Adare, the problem is clear: a tourist bus has broken down. But here is our destination, the Dunraven Arms Hotel! A wedding is underway and we must park far, far in the back.

The hotel is a delight. Quaint as all get out, but comfortable and stylish at the same time. Our room is beautiful, with a four poster bed and a large window onto a lovely courtyard garden.

We walk through the gardens and the swallows are everywhere, swooping and darting and catching dinner. It seems there is a nest in every awning and eave.

Adare is often called "the prettiest village in Ireland" and with good reason. Thatched houses line the street, centuries-old abbey and friaries are just off the road, and the thousand year old castle is just across the river.

We walk up and down the town streets, buying some fudge from a local creamery, and later having a Beamish and a Caledonia Smooth to accompany our sandwich and soup at Auntie Lena's Pub.

After our meal, we walk through the Adare Town Park, which is far too scenic and pleasant for that simple name, then return to the hotel to unwind, check email, and plan the next day.

Categories: FLOSS Project Planets

Understanding Icons: Participate in survey ’7 of 9′ (or more)

Planet KDE - Sat, 2014-08-30 14:09

With this next icon test we take a look at the Elementary icon set. Please, again, participate and help us to learn more about the usability of icon design.

Keep on reading: Understanding Icons: Participate in survey ’7 of 9′ (or more)

Categories: FLOSS Project Planets

John Goerzen: 2AM to Seattle

Planet Debian - Sat, 2014-08-30 12:11

Monday morning, 1:45AM.

Laura and I walk into the boys’ room. We turn on the light. Nothing happens. (They’re sound sleepers.)

“Boys, it’s time to get up to go get on the train!”

Four eyes pop open. “Yay! Oh I’m so excited!”

And then, “Meow!” (They enjoy playing with their stuffed cats that Laura got them for Christmas.)

Before long, it was out the door to the train station. We even had time to stop at a donut shop along the way.

We climbed into our family bedroom (a sleeping car room on Amtrak specifically designed for families of four), and as the train started to move, the excitement of what was going on crept in. Yes, it’s 2:42AM, but these are two happy boys:

Jacob and Oliver love trains, and this was the beginning of a 3-day train trip from Newton to Seattle that would take us through Kansas, Colorado, the Rocky Mountains of New Mexico, Arizona, Los Angeles, up the California coast, through the Cascades, and on to Seattle. Whew!

Here we are later that morning before breakfast:

Here’s our train at a station stop in La Junta, CO:

And at the beautiful small mountain town of Raton, NM:

Some of the passing scenery in New Mexico:

Through it all, we found many things to pass the time. I don’t think anybody was bored. I took the boys “exploring the train” several times — we’d walk from one end to the other and see what all was there. There was always the dining car for our meals, the lounge car for watching the passing scenery, and on the Coast Starlight, the Pacific Parlor Car.

Here we are getting ready for breakfast one morning.

Getting to select meals and order in the “train restaurant” was a big deal for the boys.

Laura brought one of her origami books, which even managed to pull the boys away from the passing scenery in the lounge car for quite some time.

Origami is serious business:

They had some fun wrapping themselves around my feet and challenging me to move. And were delighted when I could move even though they were trying to weight me down!

Several games of Uno were played, but even those sometimes couldn’t compete with the passing scenery:

The Coast Starlight features the Pacific Parlor Car, which was built over 50 years ago for the Santa Fe Hi-Level trains. They’ve been updated; the upper level is a lounge and small restaurant, and the lower level has been turned into a small theater. They show movies in there twice a day, but most of the time, the place is empty. A great place to go with little boys to run around and play games.

The boys and I sort of invented a new game: roadrunner and coyote, loosely based on the old Looney Tunes cartoons. Jacob and Oliver would be roadrunners, running around and yelling “MEEP MEEP!” Meanwhile, I was the coyote, who would try to catch them — even briefly succeeding sometimes — but ultimately fail in some hilarious way. It burned a lot of energy.

And, of course, the parlor car was good for scenery-watching too:

We were right along the Pacific Ocean for several hours – sometimes there would be a highway or a town between us and the beach, but usually there was nothing at all between us and the coast. It was beautiful to watch the jagged coastline go by, to gaze out onto the ocean, watching the birds — apparently so beautiful that I didn’t even think to take some photos.

Laura’s parents live in California, and took a connecting train. I had arranged for them to have a sleeping car room near ours, so for the last day of the trip, we had a group of 6. Here are the boys with their grandparents at lunch Wednesday:

We stepped off the train in Seattle into beautiful King Street Station.

Our first day in Seattle was a quiet day of not too much. Laura’s relatives live near Lake Washington, so we went out there to play. The boys enjoyed gathering black rocks along the shore.

We went blackberry picking after that – filled up buckets for a cobbler.

The next day, we rode the Seattle Monorail. The boys have been talking about this for months — a kind of train they’ve never been on. That was the biggest thing in their minds that they were waiting for. They got to ride in the very front, by the operator.

Nice view from up there.

We walked through the Pike Market — I hadn’t been in such a large and crowded place like that since I was in Guadalajara:

At the Seattle Aquarium, we all had a great time checking out all the exhibits. The “please touch” one was a particular hit.

Walking underneath the salmon tank was fun too.

We spent a couple of days doing things closer to downtown. Laura’s cousin works at MOHAI, the Museum of History and Industry, so we spent a morning there. The boys particularly enjoyed the old periscope mounted to the top of the building, and the exhibit on chocolate (of course!)

They love any kind of transportation, so of course we had to get a ride on the Seattle Streetcar that comes by MOHAI.

All weekend long, we had been noticing the seaplanes taking off from Lake Washington and Lake Union (near MOHAI). So finally I decided to investigate, and one morning while Laura was doing things with her cousin, the boys and I took a short seaplane ride from one lake to another, and then rode every method of transportation we could except for ferries (we did that the next day). Here is our Kenmore Air plane:

The view of Lake Washington from 1000 feet was beautiful:

I think we got a better view than the Space Needle, and it probably cost about the same anyhow.

After splashdown, we took the streetcar to a place where we could eat lunch right by the monorail tracks. Then we rode the monorail again. Then we caught a train (it went underground a bit so it was a “subway” to them!) and rode it a few blocks.

There is even scenery underground, it seems.

We rode a bus back, and saved one last adventure for the next day: a ferry to Bainbridge Island.

Laura and I even got some time to ourselves to go have lunch at an amazing Greek restaurant to celebrate a year since we got engaged. It’s amazing to think that, by now, it’s only a few months until our wedding anniversary too!

There are many special memories of the weekend I could mention — visiting with Laura’s family, watching the boys play with her uncle’s pipe organ (it’s in his house!), watching the boys play with their grandparents, having all six of us on the train for a day, flying paper airplanes off the balcony, enjoying the cool breeze on the ferry and the beautiful mountains behind the lake. One of my favorites is waking up to high-pitched “Meow? Meow meow meow! Wake up, brother!” sorts of sounds. There was so much cat-play on the trip, and it was cute to hear. I have the feeling we won’t hear things like that much more.

So many times on the trip I heard, “Oh dad, I am so excited!” I never get tired of hearing that. And, of course, I was excited, too.

Categories: FLOSS Project Planets

Joachim Breitner: DebConf 14

Planet Debian - Sat, 2014-08-30 10:10

I’m writing this blog post on the plain from Portland towards Europe (which I now can!), using the remaining battery live after having watched one of the DebConf talks that I missed. (It was the systemd talk, which was good and interesting, but maybe I should have watched one of the power management talks, as my battery is running down faster than it should be, I believe.)

I mostly enjoyed this year’s DebConf. I must admit that I did not come very prepared: I had neither something urgent to hack on, nor important things to discuss with the other attendees, so in a way I had a slow start. I also felt a bit out of touch with the project, both personally and technically: In previous DebConfs, I had more interest in many different corners of the project, and also came with more naive enthusiasm. After more than 10 years in the project, I see a few things more realistic and also more relaxed, and don’t react on “Wouldn’t it be cool to have crazy idea” very easily any more. And then I mostly focus on Haskell packaging (and related tooling, which sometimes is also relevant and useful to others) these days, which is not very interesting to most others.

But in the end I did get to do some useful hacking, heard a few interesting talks and even got a bit excited: I created a new tool to schedule binNMUs for Haskell packages which is quite generic (configured by just a regular expression), so that it can and will be used by the OCaml team as well, and who knows who else will start using hash-based virtual ABI packages in the future... It runs via a cron job on people.debian.org to produce output for Haskell and for OCaml, based on data pulled via HTTP. If you are a Debian developer and want up-to-date results, log into wuiet.debian.org and run ~nomeata/binNMUs --sql; it then uses the projectb and wanna-build databases directly. Thanks to the ftp team for opening up incoming.debian.org, by the way!

Unsurprisingly, I also held a talk on Haskell and Debian (slides available). I talked a bit too long and we had too little time for discussion, but in any case not all discussion would have fitted in 45 minutes. The question of which packages from Hackage should be added to Debian and which not is still undecided (which means we carry on packaging what we happen to want in Debian for whatever reason). I guess the better our tooling gets (see the next section), the more easily we can support more and more packages.

I am quite excited by and supportive of Enrico’s agenda to remove boilerplate data from the debian/ directories and relying on autodebianization tools. We have such a tool for Haskell package, cabal-debian, but it is unofficial, i.e. neither created by us nor fully endorsed. I want to change that, so I got in touch with the upstream maintainer and we want to get it into shape for producing perfect Debian packages, if the upstream provided meta data is perfect. I’d like to see the Debian Haskell Group to follows Enrico’s plan to its extreme conclusion, and this way drive innovation in Debian in general. We’ll see how that goes.

Besides all the technical program I enjoyed the obligatory games of Mao and Werewolves. I also got to dance! On Saturday night, I found a small but welcoming Swing-In-The-Park event where I could dance a few steps of Lindy Hop. And on Tuesday night, Vagrant Cascadian took us (well, three of us) to a blues dancing night, which I greatly enjoyed: The style was so improvisation-friendly that despite having missed the introduction and never having danced Blues before I could jump right in. And in contrast to social dances in Germany, where it is often announced that the girls are also invited to ask the boys, but then it is still mostly the boys who have to ask, here I took only half a minute of standing at the side until I got asked to dance. In retrospect I should have skipped the HP reception and went there directly...

I’m not heading home at the moment, but will travel directly to Göteborg to attend ICFP 2014. I hope the (usually worse) west-to-east jet lag will not prevent me from enjoying that as much as I could.

Categories: FLOSS Project Planets

Ian Ozsvald: Slides for High Performance Python tutorial at EuroSciPy2014 + Book signing!

Planet Python - Sat, 2014-08-30 06:06

Yesterday I taught an excerpt of my 2 day High Performance Python tutorial as a 1.5 hour hands-on lesson at EuroSciPy 2014 in Cambridge with 70 students:

We covered profiling (down to line-by-line CPU & memory usage), Cython (pure-py and OpenMP with numpy), Pythran, PyPy and Numba. This is an abridged set of slides from my 2 day tutorial, take a look at those details for the upcoming courses (including an intro to data science) we’re running in October.

I also got to do a book-signing for our High Performance Python book (co-authored with Micha Gorelick), O’Reilly sent us 20 galley copies to give away. The finished printed book will be available via O’Reilly and Amazon in the next few weeks.

If you want to hear about our future courses then join our low-volume training announce list. I have a short (no-signup) survey about training needs for Pythonistas in data science, please fill that in to help me figure out what we should be teaching.

I also have a further survey on how companies are using (or not using!) data science, I’ll be using the results of this when I keynote at PyConIreland in October, your input will be very useful.

Here are the slides (License: CC By NonCommercial), there’s also source on github:

Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.
Categories: FLOSS Project Planets

Claus Ibsen: Apache Camel 2.14 on the way

Planet Apache - Sat, 2014-08-30 04:37
This is just a little blog update to say that we are working on the final touches for the next release of Apache Camel which is version 2.14.

We have a discussion thread on the Apache Camel mailing list where you can provide feedback and follow the process.

There is a lot of great stuff in this release which I frankly need to blog about when I get a chance and spirt to write.

After getting some bug fixes into the Camel codebase today I plan to work on finishing the codehale metrics UI in hawtio. This allows Camel end users to easily add additional metrics about their route performances, using the metrics from codehale. We then use a metrics javascript ui, which we then integrate in hawtio, which makes it possible to render live charting which is shown in a screenshot below:

Camel routes using codehale metrics statistics, rendered in real timeAnother great new functionality in Camel 2.14, is the new DSL for rest services, though that warrants a blog entry by itself. And we also integrated with Swagger, so these REST services can provide an API out of the box.

The Apache Camel 2.14 release is expected out in September 2014. We will try to see if we are ready next week to start cutting the release.

If you have anything you want included or fixed then its time now to speak up.

Categories: FLOSS Project Planets

Gergely Nagy: Happy

Planet Debian - Sat, 2014-08-30 01:16

For the past decade or so, I wasn't exactly happy. I struggled with low self esteem, likely bordered on depression at times. I disappointed friends, family and most of all, myself. There were times I not only disliked the person I was, but hated it. This wasn't healthy, nor forward-looking, I knew that all along, and that made the situation even worse. I tried to maintain a more enthusiastic mask, pretended that nothing's wrong. Being fully aware that there actually is nothing terribly wrong, while still feeling worthless, just added insult to injury.

In the past few years, things started to improve. I had a job, things to care about, things to feel passionate about, people around me who knew nothing about the shadows on my heart, yet still smiled, still supported me. But years of self loathing does not disappear overnight.

Then one day, some six months ago, my world turned upside down. Years of disappointment, hate and loathing - poof, gone. Today, I'm happy. This is something I have not been able to tell myself in all honesty in this century yet (except maybe for very brief periods of time, when I was happy for someone else).

A little over six months ago, I met someone, someone I could open up to. I still remember the first hour, where we talked about our own shortcomings and bad habits. At the end of the day, when She ordered me a crazy-pancake (a pancake with half a dozen random fillings), I felt happy. She is everything I could ever wish for, and more. She isn't just the woman I love, with whom I'll say the words in a couple of months. She's much more than a partner, a friend a soul-mate combined in one person. She is my inspiration, my role model and my Guardian Angel too.

I no longer feel worthless, nor inadequate. I am disappointed with myself no more. I do not hate, I do not loathe, and past mistakes, past feelings seem so far away! I can share everything with Her, She does not judge, nor condemn: she supports and helps. With Her, I am happy. With Her, I am who I wanted myself to be. With Her, I am complete.

Thank You.

Categories: FLOSS Project Planets

Matthew Palmer: Chromium tabs crashing and not rendering correctly?

Planet Debian - Fri, 2014-08-29 22:45

If you’ve noticed your chrome/chromium on Linux having problems since you upgraded to somewhere around version 35/36, you’re not alone. Thankfully, it’s relatively easy to workaround. It will hit people who keep their browser open for a long time, or who have lots of tabs (or if you’re like me, and do both).

To tell if you’re suffering from this particular problem, crack open your ~/.xsession-errors file (or wherever your system logs stdout/stderr from programs running under X), and look for lines that look like this:

[22161:22185:0830/124533:ERROR:shared_memory_posix.cc(231)] Creating shared memory in /dev/shm/.org.chromium.Chromium.gFTQSy failed: Too many open files

And

[22161:22185:0830/124601:ERROR:host_shared_bitmap_manager.cc(122)] Cannot create shared memory buffer

If you see those errors, congratulations! The rest of this blog post will be of use to you.

There’s probably a myriad of bugs open about this problem, but the one I found was #367037: Shared memory-related tab crash. It turns out there’s a file handle leak in the chromium codebase somewhere, relating to shared memory handling. There’s no fix available, but the workaround is quite simple: increase the number of files that processes are allowed to have open.

System-wide, you can do this by creating a file /etc/security/limits.d/local-nofile.conf, containing this line:

* - nofile 65535

You could also edit /etc/security/limits.conf to contain the same line, if you were so inclined. Note that this will only take effect next time you login, or perhaps even only when you restart X (or, at worst, your entire machine).

This doesn’t help you if you’ve got Chromium already open and you’d like to stop it from crashing Right Now (perhaps restarting your machine would be a terrible hardship, causing you to lose your hard-won uptime record), then you can use a magical tool called prlimit.

The prlimit syscall is available if you’re running a Linux 2.6.36 or later kernel, and running at least glibc 2.13. You’ll have a prlimit command line program if you’ve got util-linux 2.21 or later. If not, you can use the example source code in the prlimit(2) manpage, changing RLIMIT_CPU to RLIMIT_NOFILE, and then running it like this:

prlimit <PID> 65535 65535

The <PID> argument is taken from the first number in the log messages from .xsession-errors – in the example above, it’s 22161.

And now, you can go back to using your tabs as ersatz bookmarks, like I do.

Categories: FLOSS Project Planets

Dirk Eddelbuettel: BH release 1.54.0-4

Planet Debian - Fri, 2014-08-29 21:02
Another small new release of our BH package providing Boost headers for use by R is now on CRAN. This one brings a one-file change: the file any.hpp comprising the Boost.Any library --- as requested by a fellow package maintainer needing it for a pending upload to CRAN.

No other changes were made.

Changes in version 1.54.0-4 (2014-08-29)
  • Added Boost Any requested by Greg Jeffries for his nabo package

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

Comments and suggestions are welcome via the mailing list or issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets

Bryan Pendleton: Central London in Two Days: Day Two

Planet Apache - Fri, 2014-08-29 20:47

We slept like exhausted travelers, but awoke refreshed and ready for more. Could we have a better second day? Why not?

We popped back on the Tube and exited at Westminster, hoping that by arriving early we'd have fewer crowds.

After a short wait, we are inside Westminster Abbey, where we pick up headsets for the audio tour (narrated by Jeremy Irons).

Touring Westminster Abbey is extremely fascinating. To start with, it is a beautiful building, with plenty of art and architecture to experience.

And it is packed with history and tradition. Just the day before, while viewing the Crown Jewels, we had paused to view the video of Queen Elizabeth II's coronation in 1953, so it was quite interesting to be standing in Westminster Abbey, just a few feet from the raised platform where the coronation occurred, and to see the Coronation Chair (now no longer with the Stone of Scone).

But Westminster Abbey is also overwhelming; a thousand years of history, religion, arts and sciences, warfare and country are all jumbled together and deposited upon us at once.

For myself, I was most taken by the tombs and memorials to the scientists (Newton, Darwin, John Harrison, etc.), and the artists (Chaucer, George Frideric Handel, etc.), but I do try to pay attention to the Kings and Queens.

(I must admit to be rather naive or uneducated in one respect: for some reason it didn't occur to me that everyone was buried in the church itself; I somehow had it in my head that churches were churches and graveyards were graveyards, but at any rate, as Donna says, I'm not confused about that anymore.)

After what seemed like hours of viewing the Abbey, we were both quite pleased, but also thoroughly tired, so when we found ourselves in the old 13th century monastery section of the Abbey (called the Cellarium), where to our surprise there was a delightful small cafe, we immediately stopped in.

Donna, in a moment of inspiration, spots that they are serving high tea, and so we have our tea, scones and all, and it was just superb!

After finishing our tea, we walk and walk and walk.

We cross through Parliament Square and over to St James's Park, where we walk through the park along the lake. Most of the birds are quite familiar (ducks, geese, swans, pelicans, coots, etc.) but it is fun to see them here in the heart of the city, and all the passers-by are delighted by them.

I was confused by the maps I'd read into thinking that Buckingham Palace Gardens is an open space like the other royal parks, but of course it is not, so we detour a few blocks and walk over into Belgravia, by Eaton Square and up Upper Belgrave Street.

We walk past simply dozens of foreign embassies; I have fun guessing them as we approach by spotting their flags.

Near the French Embassy in Knightsbridge, we enter Hyde Park. This can be confusing to us Yanks, because in New York, Hyde Park is a town (home of FDR), and in Illinois, Hyde Park is a town (well, a neighborhood of Chicago), but in London, Hyde Park is a park.

Anyway, we walked along the Serpentine as children played on the shore and paddleboats coasted gently through the water.

Just before we arrive at the bridge over the Serpentine, we arrive at the Lady Diana Memorial Fountain. As we near the fountain, the sun is shining and children are playing in the water.

The memorial is open, informal, and family-friendly, with little pomp or ceremony; in this way it seems to me entirely appropriate for Diana.

Just a short distance away (is this coincidence?) is another memorial to another unusual royal: the Albert Memorial. While the Victoria Memorial is right in front of Buckingham Palace, the Prince Consort's memorial is miles away in South Ken, but visually the two memorials have much in common: both are massive pedestals, adorned with statues and embellishments, and topped with gold figures.

We walk down through South Kensington past the Royal Albert Hall and by the V & A museum, its entranceway mobbed with families and street vendors.

The Royal Albert Hall, I notice, is holding its summertime series of "prom concerts," which I later look up and discover means "promenade" concerts, originally because they were held outside and the audience walked around freely, but which now means that you stand up for the show in standing-only areas!

From South Ken we ride the Underground on the District Line over to Temple. We walk up the Strand past the Royal Courts of Justice and Temple Bar, admiring the various sights along the way.

When the Strand becomes Fleet Street we stop at tiny Twining's Tea shop: 305 years in the same location!

We pass two other Christopher Wren churches, then read the magnificent St Paul's Cathedral with its enormous dome. We can just barely make out visitors enjoying the view from the tower atop the dome, but there is no time to rest, we have much more to do!

We walk down and across the delightful Millenium Bridge. A street artist has been working here this summer, painstakingly painting tiny micro-paintings between the non-skid bumps on the bridge's walkway.

He is there on the bridge now, talking with tourists and fans, and we stop and photograph some of his work.

We admire the reconstructed Globe Theatre. An afternoon show is in progress, so we can't take the tour, but we are amused when we see a number of actors come running around the side of the theatre and disappear through a back door, evidently exiting "stage left" in order to re-enter "stage right".

We walk along the south bank of the Thames. Past Blackfriar's Bridge we stop in Doggett's for a pint of Marston's and a rest. Fortified, we walk along the river until we reach the London Eye, passing a nifty statue of Laurence Olivier out in front of the National Theatre.

A large summer festival is taking place along the waterfront and the area between the Eye and County Hall is jammed.

We snap the classic pictures of the Houses of Parliament and Big Ben across the river, then cross Westminster Bridge.

We take the Tube back to the hotel and use the (free!) guest laundry while sharing a plate of fish and chips and watching the Bayonne vs Toulon rugby match on T.V.

I am baffled by Rugby.

But the fish and chips is delicious.

Categories: FLOSS Project Planets

Justin Mason: Links for 2014-08-29

Planet Apache - Fri, 2014-08-29 18:58
Categories: FLOSS Project Planets
Syndicate content