Apache CXF 2.7.4 (and 2.6.7 + 2.5.10) have been released. Users are strongly encouraged to upgrade to the latest versions, due to a critical security issue which must remain undisclosed for the moment. These latest releases pick up Apache Santuario 1.5.4 and Apache WSS4J 1.6.10. In addition to the fixes in these projects, CXF 2.7.4 contains a number of security fixes of interest.

1) WS-SecurityPolicy fixes

A large number of negative tests for WS-Security(Policy) were added to CXF to try to smoke out some remaining issues surrounding validating a request against a defined security policy. The following issues were fixed as a result:

• Layout policies "LaxTimestampFirst" and "LaxTimestampLast" were not validated correctly.
• X509Token "PKI" policies were not validated correctly.
• The "OnlySignEntireHeadersAndBody" policy was not validated correctly.
• The "ProtectTokens" policy was not validated correctly in conjunction with EndorsingSupportingTokens.
• SAML Token versions weren't validated against policy versions in certain circumstances.

2) Populating the SecurityContext from a JAAS Subject from WSS4J

An additional improvement to the WS-Security layer in CXF is that the SecurityContext is now populated from a JAAS Subject from WSS4J, if one is available. For example, if you are using a custom UsernameTokenValidator with WSS4J to validate a received UsernameToken via JAAS, and are returning the Subject (as per WSS4J's JAASUsernameTokenValidator), then CXF will attempt to extract roles from the Subject and populate the SecurityContext accordingly. The advantage of this is that a user can check the standard SecurityContext methods (e.g. "isUserInRole") to perform authorization.

This is controlled by two JAX-WS properties (see the documentation for more information):

• ws-security.role.classifier - The Subject Role Classifier to use.  If this value is not specified, then it tries to get roles using the DefaultSecurityContext in cxf-rt-core. Otherwise it uses this value in combination with the "ws-security.role.classifier.type" property to get the roles from the Subject.
• ws-security.role.classifier.type - The Subject Role Classifier Type to use. Currently accepted values are "prefix" or "classname". Must be used in conjunction with the "ws-security.role.classifier". The default value is "prefix".

3) STS fixes

The SecurityTokenService (STS) fixes contained in this release are:

• The STS Client was not always sending an "AppliesTo" address in a request to the STS, depending on how the STS Client was deployed.
• The STS Client was always using the "old" WS-Policy namespace for "AppliesTo", instead of getting the namespace from the policy.
• The STS now supports processing Claims in a request that are retrieved from a security policy as "IssuedToken/Claims". Previously, it would only issue claims that were contained in a "RequestSecurityTokenTemplate" policy. 

4) CVE-2012-5575

Finally, a note on security advisory CVE-2012-5575 was added to the CXF security advisories page. This attack exploits the fact that Apache CXF will attempt to decrypt arbitrary ciphertexts, without first checking to see if the algorithm corresponds to the given encryption algorithm defined by the WS-SecurityPolicy AlgorithmSuite definition. This can be exploited by chosen ciphertext attacks to retrieve the plaintext. Please note that this issue has been fixed since CXF 2.5.7, 2.6.4, and 2.7.1.

Blog post edited by Christian Schneider

Shows how to publish and use a simple REST and SOAP service in karaf using cxf and blueprint.

To run the example you need to install the http feature of karaf. The default http port is 8080 and can be configured using the
config admin pid "org.ops4j.pax.web". You also need to install the cxf feature. The base url of the cxf servlet is by default "/cxf".
It can be configured in the config pid "org.apache.cxf.osgi".

PersonService Example

The "business case" is to manage a list of persons. As service should provide the typical CRUD operations. Front ends should be a REST service, a SOAP service and a web UI.

The example consists of four projects

• model: Person class and PersonService interface
• server: Service implementation and logic to publish the service using REST and SOAP
• proxy: Accesses the SOAP service and publishes it as an OSGi service
• webui: Provides a simple servlet based web ui to list and add persons. Uses the OSGi service

You can find the full source on github: https://github.com/cschneider/Karaf-Tutorial/tree/master/cxf/personservice

Installation and test run

First we build, install and run the example to give an overview of what it does. The following main chapter then explains in detail how it works.

Installing Karaf and preparing for CXF

We start with a fresh Karaf 2.3.1.

• Unpack Karaf 2.3.1 from http://karaf.apache.org/index/community/download.html
• If you use Karaf 2.2.x you will have to copy etc/jre.properties.cxf to jre.properties
• Run Karaf using bin/karaf

Installing CXF

In Karaf Console run

features:chooseurl cxf 2.7.4 features:install http cxf Changes in commands for karaf 3

• features:chooseurl -> feature:repo-add
• features:install -> feature:install

Build and Test

Checkout the project from github and build using maven

> mvn clean install

Install service and ui in karaf install -s mvn:net.lr.tutorial.karaf.cxf.personservice/personservice-model/1.0-SNAPSHOT install -s mvn:net.lr.tutorial.karaf.cxf.personservice/personservice-server/1.0-SNAPSHOT install -s mvn:net.lr.tutorial.karaf.cxf.personservice/personservice-proxy/1.0-SNAPSHOT install -s mvn:net.lr.tutorial.karaf.cxf.personservice/personservice-webui/1.0-SNAPSHOT Test the service

The person service should show up in the list of currently installed services that can be found herehttp://localhost:8181/cxf/

List the known personshttp://localhost:8181/cxf/person
This should show one person "chris"

Now using a firefox extension like Poster or Httprequester you can add a person.

Send the following xml snippet:

<?xml version="1.0" encoding="UTF-8"?> <ns1:person xmlns:ns1="http://person.jms2rest.camel.karaf.tutorial.lr.net">     <id>1001</id>     <name>Christian Schneider</name>     <url>http://www.liquid-reality.de</url> </ns1:person>

with Content-Type:text/xml using PUT:http://localhost:8181/cxf/person/1001
or to this url using POST:http://localhost:8181/cxf/person

Now the list of persons should show two persons.

Test the proxy and web UI

http://localhost:8181/personui

You should see the list of persons managed by the personservice and be able to add new persons.

How it works Defining the model

The model project is a simple java maven project that defines a JAX-WS service and a JAXB data class. It has no dependencies to cxf. The service interface is just a plain java interface with the @WebService annotation.

@WebService public interface PersonService { public abstract Person[] getAll(); public abstract Person getPerson(String id); public abstract void updatePerson(String id, Person person); public abstract void addPerson(Person person); }

The Person class is just a simple pojo with getters and setters for id, name and url and the necessary JAXB annotations. Additionally you need an ObjectFactory to tell JAXB what xml element to use for the Person class.
There is also no special code for OSGi in this project. So the model works perfectly inside and outside of an OSGi container.

The service is defined java first. SOAP and rest are used quite transparently. This is very suitable to communicate between a client and server of the same application. If the service
is to be used by other applications the wsdl first approach is more suitable. In this case the model project should be configured to generate the data classes and service interface from
a wsdl (see cxf wsdl_first example pom file). For rest services the java first approach is quite common in general as the client typically does not use proxy classes anyway. Service implementation (server)

PersonServiceImpl is a java class the implements the service interface and contains some additional JAX-RS annotations. The way the class is defined allows it to implement a REST service and a SOAP service at the same time.

The server project also contains a small starter class that allows the service to be published directly from eclipse. This class is not necessary for deployment in karaf.

The production deployment of the service is done in src/main/resources/OSGI-INF/blueprint/blueprint.xml.

As the file is in the special location OSGI-INF/blueprint it is automatically processed by the blueprint implementation aries in karaf. The REST service is published using the jaxrs:server element and the SOAP service is published using the jaxws:endpoint element. The blueprint namespaces are different from spring but apart from this the xml is very similar to a spring xml.

Service proxy

The service proxy project only contains a blueprint xml that uses the CXF JAXWS client to consume the SOAP service and exports it as an OSGi Service. Encapsulating the service client as an OSGi service (proxy project) is not strictly necessary but it has the advantage that the webui is then completely independent of cxf. So it is very easy to change the way the service is accessed. So this is considered a best practice in OSGi.

See blueprint.xml

Web UI (webui)

This project consumes the PersonService OSGi service and exports the PersonServlet as an OSGi service. The pax web whiteboard extender will then publish the servlet on the location /personui.
The PersonServlet gets the PersonService injected and uses to get all persons and also to add persons.

The wiring is done using a blueprint context.

Some further remarks

The example uses blueprint instead of spring dm as it works much better in an OSGi environment. The bundles are created using the maven bundle plugin. A fact that shows how well blueprint works
is that the maven bundle plugin is just used with default settings. In spring dm the imports have to be configured as spring needs access to many implementation classes of cxf. For spring dm examples
take a look at the Talend Service Factory examples (https://github.com/Talend/tsf/tree/master/examples).

The example shows that writing OSGi applications is quite simple with aries and blueprint. It needs only 153 lines of java code (without comments) for a complete little application.
The blueprint xml is also quite small and readable.

Back to Karaf Tutorials

View Online

AWS CloudFormation provides a simple-yet-powerful way to create ‘stacks’ of Cloud resources with a single call. The stack is described in a parameterized template file; creation of the stack is a simple matter of providing stack parameters. The template includes description of resources such as instances and security groups and provides a language to describe the ordering dependencies between the resources.

CloudStack doesn’t have any such tool (although it has been discussed). I was interested in exploring what it takes to provide stack creation services to a CloudStack deployment. As I read through various sample templates, it was clear that the structure of the template imposed an ordering of resources. For example, an ‘Instance’ resource might refer to a ‘SecurityGroup’ resource — this means that the security group has to be created successfully first before the instance can be created. Parsing the LAMP_Single_Instance.template for example, the following dependencies emerge:

WebServer depends on ["WebServerSecurityGroup", "WaitHandle"] WaitHandle depends on [] WaitCondition depends on ["WaitHandle", "WebServer"] WebServerSecurityGroup depends on []

This can be expressed as a Directed Acyclic Graph — what remains is to extract an ordering by performing a topological sort of the DAG. Once sorted, we need an execution engine that can take the schedule and execute it. Fortunately for me, Ruby has both: the TSort module performs topological sorts and the wonderful Ruote workflow engine by @jmettraux. Given the topological sort produced by TSort:

["WebServerSecurityGroup", "WaitHandle", "WebServer", "WaitCondition"]

You can write a process definition in Ruote:

Ruote.define my_stack do sequence WebServerSecurityGroup WaitHandle WebServer WaitCondition end end

What remains is to implement the ‘participants‘ inside the process definition. For the most part it means making API calls to CloudStack to create the security group and instance. Here, the freshly minted CloudStack Ruby client from @chipchilders came in handy.

Stackmate is the result of this investigation — satisfyingly it is just 350 odd lines of ruby or so.

Ruote gives a nice split between defining the flow and the actual work items. We can ask Ruote to roll back (cancel) a process that has launched but not finished. We can create resources concurrently instead of in sequence. There’s a lot more workflow patterns here. The best part is that writing the participants is relatively trivial — just pick the right CloudStack API call to make.

While prototyping the design, I had to make a LOT of instance creation calls to my CloudStack installation — since I don’t have a ginormous cloud in back pocket, the excellent CloudStack simulator filled the role.

Next Steps

• As it stands today  stackmate is executed on the command line and the workflow executes on the client side (server being CloudStack). This mode is good for CloudStack developers performing a pre-checkin test or QA developers developing automated tests. For a production CloudStack however,  stackmate needs to be a webservice and provide a user interface to launch CloudFormation templates.
• TSort generates a topologically sorted sequence; this can be further optimized by executing some steps in parallel.
• There’s more participants to be written to implement templates with VPC resources
• Implement rollback and timeout

Advanced

Given ruote’s power, Ruby’s flexibility and the generality of CloudFormation templates:

• We should be able to write CloudStack – specific templates (e.g, to take care of stuff like network offerings)
• We should be able to execute AWS templates on clouds like Google Compute Engine
• QA automation suddenly becomes a matter of writing templates rather than error-prone API call sequences
• Templates can include custom resources such as 3rd party services: for example, after launching an instance, make an API call to a monitoring service to start monitoring port 80 on the instance, or for QA automation: make a call to a testing service
• Even more general purpose complex workflows: can we add approval workflows, exception workflows and so on. For example, a manager has to approve before the stack can be launched. Or if the launch fails due to resource limits, trigger an approval workflow from the manager to temporarily bump up resource limits.

This information is all covered in much more detail elsewhere on the web but for my own future reference, here’s a primer on doctypes, compatibility modes, charsets and fonts which was required to explain why certain Chinese characters weren’t showing up in IE8. Of course the best answer is that you need to have the East Asian Font pack installed and then it just works (usually) but this tends to be useful background and saves “server side” folks from a number of gotchas.

Doctypes and Compatibility Modes

• IE 7 and above has an insane array of compatibility modes which are out to get you. The most common gotcha is that it will use compatibility mode (emulating IE7) if the website is in the “intranet zone”. There’s an option to disable this somewhere in the preferences dialogs.  You wind up in the intranet zone if you’re accessing a site via any domain name that doesn’t look like a real one (e.g. http://dog/ is in the intranet zone).
• If you can avoid falling into compatibility mode, any pages including the DOCTYPE as <!DOCTYPE html> or <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> will render in standards compliant mode (where the world is as sane as it gets in web development). Go with the shorter version unless you have a reason not to.
• http://hsivonen.iki.fi/doctype/ is the gold standard for information about browser modes.

Character Sets and Fonts

• If you’re tracking down problems with foreign languages there are two major categories of problems – encoding corruption (where characters come out garbled or as ?) and missing glyphs in fonts (where characters come out as little boxes).
• Corruption is fixed by specifying the same character encoding everywhere. It is a security issue if any webpage is missing a meta tag defining the character set (smallest variant is <meta charset="UTF-8">). It must be the first tag in the <head> of the document.
• Little square boxes mean that either the font currently in use doesn’t have a glyph for that particular character and the font fallback routine was unable to find any font on the system which supports that character. 
• Browsers have a default stylesheet which is automatically applied to every page which commonly sets a specific font-family and font-size for text input elements, so adding the style body { font-family: 'Arial Unicode MS' } may get some Asian characters working in the main content but not in text boxes unless you also add input { font-family: inherit; }.

The security issue mentioned above is that any page which doesn’t define a character set but includes any form of user supplied content is vulnerable to a cross site scripting injection attack – even if the user supplied content is escaped properly, because the content may include a character that causes the browser to incorrectly switch to UCS-7 or other weird character sets and drastically change the meaning of the content on the page (hence the user content is no longer correctly escaped). There have been steps taken by modern browsers to remove this risk (including removing support for UCS-7 I believe) but its good practice to specify your charset explicitly anyway.

• Functional Reactive Programming in the Netflix API with RxJava

  Hmm, this seems nifty as a compositional building block for Java code to enable concurrency without thread-safety and sync problems.

  Functional reactive programming offers efficient execution and composition by providing a collection of operators capable of filtering, selecting, transforming, combining and composing Observable’s. The Observable data type can be thought of as a “push” equivalent to Iterable which is “pull”. With an Iterable, the consumer pulls values from the producer and the thread blocks until those values arrive. By contrast with the Observable type, the producer pushes values to the consumer whenever values are available. This approach is more flexible, because values can arrive synchronously or asynchronously.

  (tags: concurrency java jvm threads thread-safety coding rx frp fp functional-programming reactive functional async observable)

• You probably shouldn’t use a spreadsheet for important work

  Daniel Lemire comments on the recent cases of bugs in spreadsheets causing major impact:

  There are several critical problems with a tool like Excel that need to be widely known: * Spreadsheets do not support testing. For anything that matters, you should validate and test your code automatically and systematically; * Spreadsheets make code reviews impractical. To visually inspect the code, you need to click and each and every cell. In practice, this means that you cannot reasonably ask someone to read over your formulas to make sure that there is no mistake; * Spreadsheets encourage redundancies. Spreadsheets encourage copy-and-paste. Though copying and pasting is sometimes the right tool, it also creates redundancies. These redundancies make it very difficult to update a spreadsheet: are you absolutely sure that you have changed the formula throughout? Agreed on all three, particularly on the impossibility of testing. IMO, everyone who may be in a job where automation via spreadsheet is likely, needs training in SDE fundamentals: unit testing, the important of open source and open data for reproducibility, version control, and code review. We are all computer scientists now.

  (tags: spreadsheets excel coding errors bugs testability unit-testing testing quality sde sde-fundamentals dry)

• Log4j2 Asynchronous Loggers for Low-Latency Logging – Apache Log4j 2

  implemented using the LMAX Disruptor library — very impressive performance figures. I presume in real-world usage, these latencies are dwarfed by hardware costs, though

  (tags: disruptor coding java log4j logging async performance)

Self portrait in an old house in central Havana©James Duncan Davidson

Going to photograph a place like Cuba is not a simple exercise and it’s full of questions. As an American subject to the rules of the US government’s embargo against Cuba, one of the first questions is how do you get there, legally? There’s a variety of ways, including the person to person cultural exchanges such as the ones organized by Santa Fe Photographic Workshops. You could also go illegally via Canada, Mexico, or the Bahamas, but then you’re flying without much of a net. Europeans, of course, don’t have to deal with a prohibition on travel and are there in droves. In fact, my friends in Europe didn’t get what a big deal it was to go until I explained the situation in detail. And then they still look at me funny with a expression that says: “Really? Come on… You must be kidding.”

After simple logistics, however, the questions become nuanced and tend to group around motivation. Why even go? What are you hoping to get out of the experience? What do you hope that the locals will get out of meeting you? Will going support or defend those currently in power and how? Does going somehow increase the problems of the locals or might it actually help in some small way? These are all questions I’ve faced before in my travels—most notably when I went to Myanmar (Burma) over a year ago.

There aren’t simple answers, of course. For some—especially for those in the Cuban American community who are fairly unequivocal in their continued support of the embargo—there is absolutely no excuse for traveling to Cuba. It’s simply seen as encouraging the policies of the Castro regime by supporting them monetarily with the money made at the hotels and restaurants. It’s the same sort of argument that the Burmese opposition used to make in their call for travelers to boycott Myanmar—a stance they’ve since reversed with a new embrace a responsible form of tourism to encourage understanding between people.

Cuban photographer Ramsés H. Batista in action in the Cuban countryside©James Duncan Davidson Speaking with Leysis Quesada Vera over coffee about being a photographer in Cuba©James Duncan Davidson

Our group had hundreds of small interactions with Cubans during our week. Sometimes, it was a simple exchange of a smile. Many times, it was part of an attempt to sell a taxi ride or a cigar, inevitably starting with a “Hello my friend, where are you from?” but which sometimes evolved into a discussion about current politics or even an occasional personal dissertation of a personal history. More than one of these evolved into a long discussion. The most striking of our interactions to me, however, were the longer term ones we had with the two local Cuban photographers who accompanied us for the week: Ramsés H. Batista and Leysis Quesada.

If there’s the start of a positive entrepreneur class in Cuba—one which builds a life based on creation instead of simply taking advantage of some non-renewable or agricultural resource that can be easily exploited—these two are definitely in it. Ramsés is in the process of opening a studio. Leysis has international exhibitions of her work. Both have made and published photographs documenting day to day life in Cuba. They work within the boundaries of the system—one of the ways they make money is by helping groups like ours—but they’re part of a group of people that is using each expansion of the boundaries to build for their future.

We learned a lot from them, and I’m grateful for what they taught me about Cuban culture and the realities of being a photographer there right now. I’m pretty sure it went both ways, too. Our group was composed of professional photographers and several brilliant technologists. This lead to many great discussions between Cubans and Americans over late night dinners that covered all sorts of topics worthy of continuing discussion.

Dancers in a studio in central Havana©James Duncan Davidson Dancers checking out an iPhone after rehearsal©James Duncan Davidson

Of course, emerging entrepreneurs like Ramsés and Leysis are very much a super minority in Cuba. But almost every other Cuban with whom we interacted with was also looking forward to the future in one way or another—the younger generations being the most eager, as you might expect. We saw a surprising number of iPhones and other smart phones in use by Cubans, even though most didn’t seem to be actively online. I snuck a look at as many screens as I could and only one or two had any kind of data connectivity—and those only at GPRS speed. But people are managing to find their way and when data access does become more common place, they’re going to take full advantage of what it can do for them.

The more I travel the world, the more I believe that the most important lesson that comes from it is that the more people interact with each other, the better. Of course, not all of those interactions will be good ones. It’d be beyond naïve to expect that and certainly there’s a dark side to Cuba—just as there is in many countries. But when it’s done constructively, interaction can help tear down barriers and help people on both sides understand each other as humans instead of imagining them as what propaganda could lead you to believe.

Not to get too all philosophical here but, for me, I think one of the things that drives my travel to places like Cuba—and certainly something I became that much more aware of during this trip—is the process of tackling the “Otherness” that still drives much of the fundamentals of how our world operates. I might not be able to do much but showing up, having a few great interactions that hopefully help out a few Cubans as they build towards a different future, and then bringing home and sharing some stories—especially with everyone who sees them here on my blog—is my way of doing at least a little bit. Is it enough? Not hardly. But it’s a start.

Two young women walking along Paseo del Prado©James Duncan Davidson

There's a great meme going around about geeks and repetitive tasks. Because geeks will often get annoyed at the effort of doing something manually, they often decide to find a way to automate it – which usually involves a lot more effort than doing it the one time but "geeks win, eventually" because they save time in the long run.

But in the long run we're all dead. Then what? Who knows how to run your script? What happens when it needs to be maintained? As Jon Udell points out, it's really not a contest, it's a process, and non-geeks can play too. Which is why you should also write it down if you're going to do it more than two times.

OK, "doing it more than two times" is a huge generalization. What I mean more specifically is:

• If you're in a team environment or doing work that will keep cropping up.
• If you're doing a task that is non-obvious and/or has a complicated series of steps that is non-obvious to people who are not you.
• If you're in any kind of critical path that would block shipping or operations if you aren't there to do the magical things you do.
• If you want to reduce your project or organization's Bus Factor (help other people become proficient).
• If you want to better understand what you do and how you can improve it.

Then you need to take a step back and document the things that you do on a regular basis, because it will help your teammates and (most likely) even you when you come back to a task that you haven't done for a long time.

Naturally, I'm thinking of this in terms of a project like CloudStack where documentation is vitally important. The success of a distributed team depends a great deal on good documentation.

An Example

Yesterday I spent the better part of the day doing something that, by rights, should have take 30 to 60 minutes – depending on the whims of the bandwidth gods while copying up docs to the server.

Unfortunately, what I was working on was not well-documented. This isn't surprising, it's something we've only had to do as a project twice and the first time was the "let's figure out how to do this" run. I captured some of the documentation during the second run, but not in enough detail and missed one step that wound up setting me back by a big chunk of time.

How this could have been avoided: The second time we did this, as a project, it should have been well documented and put up on the wiki.

Getting it Down

While you're working, keep a scratch (text) file open at all times. Shell history is nice, but it's often hard to decipher after the fact, especially if you have a lot of trial and error going on.

When you run a command or do something that works, put it in the file. Even if you do nothing more than make public a list of steps that worked for you, it's light years ahead of having to start from scratch.

After you've done $task, take a few minutes to brush up the list you've created and (if necessary) fill in any gaps like the requirements needed to run the commands you've used, the files you need, where to get source – whatever. Just think about how you got to the state you were in before Step 1 that may not be obvious to anyone else.

Plain text is absolutely fine, and usually preferred.

Getting it Out

If your project uses a wiki or something like that, now's a good time to put it in the appropriate place in the wiki. Note that putting documentation in an obvious place is sometimes as important as creating the documentation in the first place. A hidden README file in a disused directory on SVN isn't much help if your project usually looks to the Web for its docs.

You can, and also should, put it up on a blog. This is not a natural impulse for most hacker types, but more's the pity that it isn't. First, you can get feedback that way from people you've never even heard of. You may take it with a grain of salt, but you may also learn something you never would have learned any other way from someone who is an expert at what you're an neophyte at. (Note, this can also serve as a resume addition of sorts that demonstrates to employers that you are capable of putting words on paper, and that you're constantly learning.)

Finally, assuming you're doing it with a project or organization: announce your new docs on the appropriate mailing list. This way anyone who has an interest can benefit from your new docs, and you also can enjoy a little peer review. (Note, make a practice of thanking people if you see this happening, as a little encouragement is nice and shows that – yes, someone actually just received the message in a bottle that was just sent out.)

Share Your Brain

If you're the type of person who'd rather automate a task than repeat it, then you should also think about helping others (and future you) avoid having to repeat learning how to do something you've already figured out. The extra time that it takes to document something is no different than the extra time it takes to write a script; it's saving work in the future by doing a little extra now.

I've never quite gotten the hang of Subversion (SVN). Most of the stuff we do on CloudStack is in a git repository, but the Website is managed with the Apache CMS and stored in SVN.

While working on the 4.0.2 docs release yesterday, I found that I didn't quite have all the steps worked out for committing docs to the Website. (We've only had two releases so far, and David Nalley took care of building and releasing the final docs on those.)

To make a long, frustrating, story a lot shorter: I found an occasion to need to know how to do an SVN revert and realized it's not just like git when it comes to doing a revert. Git makes doing a revert pretty damn easy, whether you're reverting a local change or a change you've already committed to the main repo. Subversion also makes doing a revert easy locally:

svn revert /path/to/filename

That's simple enough. But what if you've already sent your changes to the main repo? Then it's a little trickier, but not impossible.

• Get the revision number of your commit. Sooner is better (I'm not sure how easy this would be if you're trying to revert a change a month later if there's been a lot of changes around it in the meantime.)
• Do an svn update: svn update
• Then you're going to (and this isn't terribly intuitive) do an svn merge to walk it back: svn merge -c -XXXXXXXX https://svn.apache.org/repos/asf/cloudstack (Naturally, you're going to want to replace the URL to the repo with your own, rather than CloudStack's. And replace XXXXXXXX with the number of your own commit.) The -c -XXXXXXXX means change in reverse.
• Now, do an svn status to make sure that it's properly undone what you wanted undone. You can do an svn diff (svn di) as well if you want detailed changes.
• Finally, you'll go ahead and commit the changes if you're convinced all is well: svn commit -m "Oops"

(The oops is actually optional.)

Folks who work with SVN regularly no doubt know all this, but I know there are plenty of folks like me who touch SVN infrequently and might run into trouble.

• Archiving Gmail to Evernote

  Google Drive and GMail have a built-in scripting engine. I had no idea

  (tags: gmail evernote archival scripting coding hacks google-drive)

• The Why

  How the Irish media are partly to blame for the catastrophic property bubble, from a paper entitled _The Role Of The Media In Propping Up Ireland’s Housing Bubble_, by Dr Julien Mercille, in the _Social Europe Journal_:

  “The overall argument is that the Irish media are part and parcel of the political and corporate establishment, and as such the news they convey tend to reflect those sectors’ interests and views. In particular, the Celtic Tiger years involved the financialisation of the economy and a large property bubble, all of it wrapped in an implicit neoliberal ideology. The media, embedded within this particular political economy and itself a constitutive element of it, thus mostly presented stories sustaining it. In particular, news organisations acquired direct stakes in an inflated real estate market by purchasing property websites and receiving vital advertising revenue from the real estate sector. Moreover, a number of their board members were current or former high officials in the finance industry and government, including banks deeply involved in the bubble’s expansion.”

  (tags: economics irish-times ireland newspapers media elite insiders bubble property-bubble property celtic-tiger papers news bias)

• transparent DNS proxies

  Ugh. low-end ISPs MITM’ing DNS queries:

  Some ISP’s are now using a technology called ‘Transparent DNS proxy’. Using this technology, they will intercept all DNS lookup requests (TCP/UDP port 53) and transparently proxy the results. This effectively forces you to use their DNS service for all DNS lookups. If you have changed your DNS settings to an open DNS service such as Google, Comodo or OpenDNS expecting that your DNS traffic is no longer being sent to your ISP’s DNS server, you may be surprised to find out that they are using transparent DNS proxying. (via Nelson)

  (tags: via:nelson dns isps proxying mitm phorm attacks)

• BitTorrent’s Secure Dropbox Alternative Goes Public

  As kragen says, ‘a decentralized way to sync a folder of large files, using BitTorrent instead of an untrustworthy central server’. Windows, OSX, and Linux supported

  (tags: bittorrent dropbox cloud storage filesharing sharing sync synchronization)

Apache CloudStack graduated to become a top level project at the ASF last month, and a number of community members have been blogging about their experience. CloudStack started with a company called Cloud.com, was purchased by Citrix, and then was submitted to the Apache Incubator last year to now become a full Apache project.

Along with the great CloudStack software that Apache can now provide that allows you to manage your own public or private IaaS clouds, the Apache community has gained a great new community of committers, users, and PMC members.

In reading the several blog posts by key CloudStack contributors, I reminded myself that kudos were in order as well.

Having watched Citrix bring their code and developers to the Apache Incubator, and having watched (and commented on and answered many questions from!) the podling as it grew it’s community and graduated, I’ve been struck by how well the core Citrix contributors and their many other participants really took to the Apache Way.

Both Citrix as an organization (which employs some of the CloudStack committers), and especially the many contributors to the CloudStack project took the incubation process seriously, and have really gone above and beyond to ensure their podling proposal and march to graduation have been about Apache CloudStack, as well as being about an inclusive and meritocratic project.

The desire to get things “right” at Apache was clear in everything the CloudStack community did, and the end result looks to be an incredibly strong project that’s quickly gathering developers from a wide variety of vendors and users. Part of this growth is about the great technology; but a lot is due to the helpful and welcoming face that the CloudStack committers put on their project.

We’ve had a lot of great projects, and many great communities come to the Apache Incubator; there are a lot of people to thank across the tremendous spectrum of no-charge software that the ASF provides for the public good. But I just wanted to mention the extra effort the CloudStack community put into fully embracing the Apache Way. Good job, and thanks!

Maybe not so well known feature in Wicket is that the form component validators are actually Behaviors, or if they are not then a special adapter is used, so at the end they are stored as a behavior in the component. Being a behavior gives the validator the chance to be notified when the respective [...]

I've been just unbelievably busy of late. For one thing, we've got a big event this week at work, which is keeping me quite occupied.

But I did finally find 30 minutes to read through the nice article in this month's Vanity Fair about that other software company founded by a Harvard drop-out: Facebook Leans In.

If you've seen The Social Network or watched the 60 Minutes episode or even if you've just been anywhere but buried in a cave above the Arctic Circle, you probably know a lot of this already.

But the article is well-written, and well-researched, and worth reading if you're at all interested about the way that technology, culture, and business are commingling.

A few of my favorite bits:

• A nice look back at how the advertising industry has undergone a series of technology-driven upheavals: Month after month, the technology’s popularity grew astronomically. In just one year, the number of users of the free service exploded by 2,500 percent, but still, no one could quite figure out how to make money from this gigantic audience. We’re talking not about Facebook here but about radio, which, at first, like social networks, seemed destined to be a financial flop. And in that tale—and in the history of advertising, from newspapers to Google—lie lessons to remember when considering the prospects for Facebook.
• A succinct summation of how advertising adapted to the Internet, and vice versa: companies had access for the first time to people on the verge of buying, with ads displayed only after users signaled their interest in particular products by the words in their searches. Click-throughs, pay-per-click, and marketing based on someone’s intent to buy became widely accepted as the long-hidden secrets that had cracked the code
• A great sound bite about how Facebook once again stood the world on its head: There was no click leading to a quick purchase, as with Google, or a push to buy a particular product, as with broadcast commercials. “There was a period when all the [chief marketing officers] said, ‘If you can’t be TV, can you at least be search?,’ because they understood that,” Sandberg says. “But we’re not search, and we’re not TV. We’re a third thing.”
• A solid illustration of that tangling of technology, culture, and business that I mentioned earlier: entire divisions have been created within companies with the job of interacting with customers on Facebook. For example, L’Oréal, the cosmetics company, has a staff of 400 people who post content on Facebook every day, according to Marc Menesguen, the company’s chief marketing officer. “It’s a lot of work and requires a lot of commitment,” he says.
• And, although it's just a few words in a long article, a peek at the thought process of the man behind it all: “The way we look at it is that, if we are doing our job well, then people will come to Facebook to consume a lot of content,” he said. “If people don’t connect to the advertising content, then it’s not good for anyone. It’s not good for the people using Facebook, it’s not good for the advertisers, and then ultimately we don’t make money.”

Zuckerberg's assessment reminds me of the words of Engine Charlie, who once said

for years I thought what was good for our country was good for General Motors, and vice versa Our modern Internet behemoths (Google, Amazon, Facebook, Netflix, Twitter, ...) are reshaping the world: The future is already here – it's just not evenly distributed.

It's important to pay attention to the perspective that Eichenwald's story in Vanity Fair brings, because the changing of the world is not simply due to technology; it's more complicated than that.

But that's all about that for now; big day tomorrow, and so to bed...

• DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second

  250 million tweets per day, 30-node HBase cluster, 400TB of storage, Kafka and 0mq. This is from 2011, hence this dated line: ‘for a distributed application they thought AWS was too limited, especially in the network. AWS doesn’t do well when nodes are connected together and they need to talk to each other. Not low enough latency network. Their customers care about latency.’ (Nowadays, it would be damn hard to build a lower-latency network than that attached to a cc2.8xlarge instance.)

  (tags: datasift architecture scalability data twitter firehose hbase kafka zeromq)

• Breaking the 1000 ms Time to Glass Mobile Barrier [slides]

  Great presentation from Google on HTML5 CSS+JS render speed, 3G/4G network latency, etc. (via John G)

  (tags: google slides 3g 4g lte networking telcos telecom css js html5 web via:jg)

• Lucene 4 – Revisiting Problems For Speed [slides]

  a Presentation from Simon Willnauer on optimization work performed on Lucene in 2011. The most interesting stuff here is the work done to replace an O(n^2) FuzzyQuery fuzzy-match algorithm with a FSM trie is extremely cool — benchmarked at 214 times faster!

  (tags: benchmarks slides lucene search fuzzy-matching text-matching strings algorithms coding fsm tries)

• Microsoft Code Digger extension

  Miguel de Icaza says it’s witchcraft — I’m inclined to agree:

  Code Digger analyzes possible execution paths through your .NET code. The result is a table where each row shows a unique behavior of your code. The table helps you understand the behavior of the code, and it may also uncover hidden bugs. Through the new context menu item “Generate Inputs / Outputs Table” in the Visual Studio editor, you can invoke Code Digger to analyze your code. Code Digger computes and displays input-output pairs. Code Digger systematically hunts for bugs, exceptions, and assertion failures.

  (tags: testing constraint-solving solver witchcraft magic dot-net coding tests code-digger microsoft)

Since I re-joined IBM in 2012, I have been involved in a few Big Data related projects. For those interested in learning more about IBM's Big Data Platform, below is an article I co-authored, that introduces you to developing a big data application for data exploration and discovery using BigIsights and Data Explorer.

If you've been following many of the early case studies around big data, you may have come to believe the saying that "you don't know what you don't know." Indeed, big data applications often focus on gleaning business insights from data that might otherwise be discarded or ignored for a variety of reasons. Increasingly, companies are looking to develop a comprehensive information management strategy that involves more than simply exploring or analyzing big data. Specifically, they want to integrate big data into their overall information management strategies alongside existing data systems, including relational DBMSes, enterprise content management systems, data warehouses, etc.

This article examines one facet of that challenge, outlining an architecture and approach for indexing big data and traditional data sources, as well as providing a web-based interface for discovering new insights across these disparate data sources. In particular, it describes how Data Explorer, a data discovery platform, can index data managed by InfoSphere BigInsights, enabling persistent forms of big data to be combined with existing enterprise data. Both Data Explorer and BigInsights are key components of IBM's big data platform, so let's start with an overview of this platform and these two key offerings. Full article available in IBM developerWorks

A month ago, March 20th to be exact, the Apache CloudStack project graduated from the Apache Incubator via a resolution of the Apache Software Foundation's board of directors. While this was an exceptionally important step for the CloudStack community, and it represents a new phase in the project's lifetime, it really means that we're just getting started.

For me, it's been a fun ride with the Apache CloudStack community over the last 12 months. I started participating in the project shortly after they entered the Apache Incubator, on May 8, 2012 to be exact. Since that time, I've grown both in my understanding of the software, as well as my understanding and interest in the community itself. Apache prides itself on being focused on the health of the communities under it's umbrella, and I've found this to be the most powerful aspect of working within Apache communities. The CloudStack community is no exception to that practice. We sometimes make decisions that may seem to make short term effeciency of a particular release or technical matter seem to be a lesser concern, but these are the right decisions to make for the long haul (and the long haul is what matters really). I'd say that it's been more than effective at helping to grow the project's participation. Just look at the graphs of activity for CloudStack pre and post ASF that Sebastian has put together to get a feel for the community growth trends and see what I mean.

I'd like to thank my fellow PMC members and committers for thinking enough of me to ask me to chair the PMC this year. I'd also like to thank the entire community of users, developers, testers and doc writers. We've come a long way since the project entered graduation (more than outsiders may realize). I've certainly learned a ton about FLOSS development and community building from this bunch of folks, and plan to do everything I can to support the project's future success. The collection of CloudStack experts and ASF members that came together around the project has been amazing. I'm also excited that we decided to make the role a "rotating" job that gets revisited annually. We should be able share the work amongst each other over time.

So what's next for the project? Well, for starters our 4.1.0 release is just around the corner, and we're already wrapping up most of the features for the following 4.2.0 release.

Oh, and just in time for the summer season, the CloudStack community is holding it's second annual CloudStack Collaboration Conference in Santa Clara, CA on June 23 through 25. The call for talks was just opened, and early bird registrations are available for attendees now.

Remember though... this is an Apache project. Decisions happen on the mailing lists, or they didn't happen. If you can't make it to this year's conference, you'll still be welcomed on the lists if you want to help in any way. Feel free to join us!

Last year, I worked on a huge redesign of Taleo's UI with HTML5, Twitter Bootstrap and CSS3. Management thought it would take 6-9 months and my colleague (Vladimir Bazarsky) and I finished it in just over 3 months. Yes, we encountered many, many cross-browser compatibility issues in the process. While in QA, we found and fixed over 750 issues. This was no small feat since the app was over 2 million lines of code and contained 1700 JSPs.

After writing about working with Bootstrap, I was contacted by my good friends, David Geary and Scott Davis to speak at the HTML5 Denver Users Group. Scott was very persuasive with his words (a.k.a. lots of trash-talking) and I chuckled as I read one of the best emails I've ever received. I replied that his strategy worked, I'd come up with a excellent topic and agreed to speak in April.

That speaking engagement was last night and you can view my presentation as an HTML5 app or on SlideShare.

Much of the Bootstrap content comes from Dan Vega. He built a Bootstrap presentation last August using the HTML5 slide template for Google I/O 2012 and put it on GitHub. In an email to the Bootstrap mailing list, he wrote "feel free to use the slide deck if you plan on telling others about this awesome product" and that's exactly what I did.

I updated all the statistics, added my redesign story, included a few slides on Scalable and Modular Architecture for CSS and beautified it with Trish's Photos. I've published the result on GitHub and encourage you to fork it.

Speaking at HTML5 Denver was a real treat. The venue, Casselman's, was awesome. It has a huge room with a proper stage, sound system and lighting. If you've done something cool with HTML5 lately, I encourage you to signup for a 10-minute lightning talk next month.

Not only was the venue great, but the cold Guinness while speaking was delicious. It was also a nice networking opportunity. I met several folks afterwards and talked about what's next for me. My contract with Taleo/Oracle ends May 31st, hence the reason "Free Agent" is listed on my LinkedIn profile. I've got a few good opportunities so far, but nothing that I've agreed to yet. I expect negotiations to heat up in the coming weeks, so please let me know if you'd like a seat at the bargaining table.

Apache Syncope can manage users on external LDAP directory server since almost the beginning, via ConnId's LDAP connector.

The release 1.1.0 Ad libitum, however, has extended the capabilities to include role provisioning: for LDAP this means the possibility to manage groups (groupOfNames, groupOfUniqueNames), besides users.

Full story »

Original post blogged on b2evolution.

The decision to have time-based releases was made early on with the Apache CloudStack project. It's fairly standard these days to go with time-based releases rather than feature-based releases, and makes a lot of sense for projects that are primarily aimed at use in corporate IT.

While coming to the decision of time-based vs. feature-based was easy, coming up with the actual length of a release cycle is trickier. Everyone wants the release yesterday, so there's a strong desire to have a shorter cycle. But a shorter cycle puts pressure on folks to actually get the release out, and a shorter cycle (especially for newer projects) seems more difficult.

We decided initially to go with four-month cycles, but [that decision is being revisited|http://markmail.org/message/3ctdwor5hfbpa3vx] right now, with a lot of the community leaning in favor of six-month cycles.

I'd be curious to hear from other projects exactly how they came to decide the length of time-based release cycles. In Linux distribution circles, six-month cycles have been popular – though it looks like Ubuntu is fiddling with that a little bit, and AFAIK, Fedora has generally come closer to seven-month cycles when all is done and said.

For the record, I don't think we've got enough data to decide that four month cycles aren't workable. I'd really like to see the community take 4.2.0 as the cycle to really refine processes, get more automated testing in place, and try to get documentation folks working together a little better before saying that six-month cycles are a better fit.

A few weeks ago, our first Red Hat JBoss Fuse release was completed. As with previous releases, this comes with a few examples that show you how to build e.g. a content-based routers or a JAX-RS web service. These examples are all very simple and focused on single technology.

In real life however, things often get quite a bit more complicated than what these examples are showing you. Based on their experience with customers, people like Charles Mouillard, Scott Cranton, Jakub Korab and Scott England-Sullivan have been creating a set of more elaborate example projects on GitHub.

FuseByExample is the name of the GitHub organisation where all these projects live. If you go there, you'll find a dozen examples that show you how to implement things like transactions using Aries' JTA support,using JPA for database access, using web sockets to build more dynamic web front-ends, ...

All of these examples have been updated to work with the new Red Hat JBoss Fuse 6.0 release so if you're taking a look at this new release, be sure to check out the stuff in FuseByExample as well to get a better idea of what is possible with this technology. And also, these are GitHub projects so feel free to create pull requests for improvements, bug fixes, ... We love contributions ;)

Binary Repository Management refcard on DZone The people at DZone were kind enough to ask me to write a refcard on Binary Repository Management a few months ago, and it’s now available for download.

I wrote about benefits and best practices when using a repository and compare the three tools in the space: Apache Archiva, Sonatype Nexus and JFrog Artifactory.