Planet Apache

Syndicate content
Updated: 1 day 2 hours ago

Sebastien Goasguen: 1 Command to Mesos with Docker Compose

Wed, 2015-03-18 05:01
If you have not tried Docker, you should. The sheer power it puts in your hands and the simplicity of the user experience will just wow you. In this post, I will show you how to start a one node Mesos setup with Docker compose.

Docker announced compose on February 26th. Compose allows you to describe a multi-container setup and manage it with one binary docker-compose. The containers and volumes combinations managed by Compose are defined in a YAML file, super easy to read and super easy to write. The UX is very similar to the Docker CLI.

When compose was released, I tried it and was a bit underwhelmed, as it is basically a relooking of Fig. This is not unexpected as Docker Inc, acquired Orchard the makers of Fig. But I was expecting more added functionality and even a tighter integration with the Docker client (something a dev branch actually prototyped), even a common release instead of a separate binary. I am sure this will come.

As I am writing the docker cookbook, I have deployed Wordpress 20 different ways, and it's getting a bit boring !  I was looking for more information on Mesos and its support for Docker, I re-read a terrific blog post that showed how to start a Mesos setup (zookeeper, master, slave, marathon framework) in 7 commands. Can't beat that.

When I re-read this post, I automatically thought this was an exciting use case for docker-compose. One YAML file to start Mesos/Zookeeper/Marathon and experiment with it. Of course I am not talking about a production multi-node setup. I am just looking at it for an easy Mesos experiment.
I will spare you the details of installing compose (just a curl away). The dockers docs are great.

So here is the YAML file describing our Mesos setup:

zookeeper:
image: garland/zookeeper
ports:
- "2181:2181"
- "2888:2888"
- "3888:3888"
mesosmaster:
image: garland/mesosphere-docker-mesos-master
ports:
- "5050:5050"
links:
- zookeeper:zk
environment:
- MESOS_ZK=zk://zk:2181/mesos
- MESOS_LOG_DIR=/var/log/mesos
- MESOS_QUORUM=1
- MESOS_REGISTRY=in_memory
- MESOS_WORK_DIR=/var/lib/mesos
marathon:
image: garland/mesosphere-docker-marathon
links:
- zookeeper:zk
- mesosmaster:master
command: --master zk://zk:2181/mesos --zk zk://zk:2181/marathon
ports:
- "8080:8080"
mesosslave:
image: garland/mesosphere-docker-mesos-master:latest
ports:
- "5051:5051"
links:
- zookeeper:zk
- mesosmaster:master
entrypoint: mesos-slave
environment:
- MESOS_HOSTNAME=192.168.33.10
- MESOS_MASTER=zk://zk:2181/mesos
- MESOS_LOG_DIR=/var/log/mesos
- MESOS_LOGGING_LEVEL=INFO
Four containers, images pulled from Docker hub, some ports exposed on the host. Some container linking and some environment variables used to configure the Mesos slave and master. One small hickup in the Slave defintion. You will see that I set the MESOS_HOSTNAME to the IP of the host. This allows me to browse the stdout and stderr of a Marathon task, otherwise we cannot reach it easily (small improvement to be done there.)

Launch this with docker-compose:

$ ./docker-compose up -d
Recreating vagrant_zookeeper_1...
Recreating vagrant_mesosmaster_1...
Recreating vagrant_marathon_1...
Recreating vagrant_mesosslave_1...
And open your browser at http://IP_HOST:5050 then follow the rest of the blog to start a task in marathon.



Bottom line, I went from '7 commands to Mesos' to '1 command to Mesos' thanks to Docker-compose and a fairly simple YAML file. Got to love it. When compose can do this across Docker hosts in a Docker Swarm started by Machine. Then the real fun will begin !
Categories: FLOSS Project Planets

Justin Mason: Links for 2015-03-17

Tue, 2015-03-17 18:58
Categories: FLOSS Project Planets

Gianugo Rabellino: Setting up FreeBSD and jails on Azure – part 1: networking

Tue, 2015-03-17 11:01

I set up this blog on Azure as an excuse to play with the new FreeBSD VM Depot image, learn more about jails and write the occasional blog post about random stuff. I took extensive notes while at it and I will be posting them here for future reference and to help the occasional search engine user.

I will skip all the clicking through that can easily get to a running FreeBSD VM in Azure. There is tons of FreeBSD documentation, including specific Azure tutorials that my team and others have written. I am lazy, so I will just point out specific Azure differences and how to take care of them.

A word of caution: please don’t consider what you read here to be authoritative. I’m doing this for fun and my free time is what it is, so don’t think I researched this stuff thoroughly. It worked for me and seems to be still working as I write this – that’s all I needed.

Let’s start with networking. Every public cloud has their own approach, and Azure is no different. Two things to remember about Azure IP management:

  1. Your VM will have an internal IP that will be translated to a public one. If you are puzzled about the number not being in your expected 192.168.0.0/172.16.0.0/10.0.0.0 ranges, know that you are looking at the wrong RFC: the IP will come from a “Carrier-Grade NAT” pool specified in RFC6598. And, unless you pay more set up a VNET, it will change between reboots. Being the cheapskate I am, I didn’t bother. But here be dragons, so stay tuned.
  2. Your external IP is again assigned dynamically, and dyndns’d to [yourmachine].cloudapp.net. Again, for an additional (and actually minimal) charge you can reserve a block of IPs, but given that the IP will be reserved for the lifetime of your deployment (that is, it will survive a reboot), you may well not need to pay a premium. YMMV.

With that in mind, I quickly stumbled into an issue. Jails on FreeBSD require all the services in the host machine to be bound to specific interfaces. That is you cannot bind to 0.0.0.0, *.*.*.* or any general address. Ignore this, you may end up like me – locked out of your machine as soon as you start the first jail (damn you syslogd!).

‘sockstat -4’ is your friend here as it will list which services are bound to which interfaces. On my vanilla install I had to change syslogd and sshd as they both are configured to listen to everything and the kitchen sink.

Syslogd was a piece of cake as long as I was fine with loopback only – it may become trickier if I need to get fancy and gather logs from the network. Add this in /etc/rc.conf

syslogd_flags="-s -b 127.0.0.1"

restart syslogd and you’re off to the races. Do not try ‘-s –b [yourhostname.cloudapp.net]’: intuitive as it might seem, the DNS lookup will give you the public IP address for your host, so you will be out of luck.

SSH was more of a pickle. The ListenAddress directive in sshd.conf will accept either an explicit IP address or 0.0.0.0 to listen to every IP on your machine: there is no way use an interface name such as hn0. You can specify a hostname, but then again a DNS lookup for hostname.cloudapp.net will give you the useless public IP address so you would be out of luck.

The cleanest hack I could think of was using /etc/dhclient-exit-hooks to update sshd_config with a new ListenAddress and restart sshd with every IP changes. The first version of the script looked quite brittle for such a delicate task, so I thought I’d leave it for a rainy day and moved on to something simpler based on only two moving parts:

  1. The internal IP will not change as long as the machine is up, so it’s reasonable to expect that adding a ListenAddress directive using the current IP will be quite stable. An invalid IP address will not prevent sshd from starting so I can live with it to take care of my port 22 to begin with.
  2. Belts and suspenders come handy when it comes to logging on your machine , so it occurred to me that I could use PF (yes, I like it better than ipfw) to bind SSH to the loopback address and redirecting incoming traffic from the public interface over there. This way, if the IP address changes, I can still connect using a different port (in the example below it’s 2222).

Simply enough:

add a ListenAddress line to /etc/sshd.conf:
ListenAddress 127.0.0.1

enable pf in /etc/rc.conf:

pf_enable=”YES”

add a redirection rule on /etc/pf.conf
#INTERFACES
ext_if="hn0"
rdr pass on $ext_if inet proto tcp to port 2222 -> 127.0.0.1 port ssh

start the service
service pf start

Restart sshd, cross your fingers, open a new terminal and try to ssh in again. Do NOT, for the love of what’s dearest to you, close the shell you’re in as it could be your only way back if you did something wrong. After testing that this setup works, I have closed the endpoint on the Azure firewall, but reopening it will be a matter of seconds in case things go south.

(For the record, things went south and I got locked out nevertheless. Twice. Stay tuned as I will make copious fun of myself in an upcoming post, together with providing a few survival tips to recover a deployment gone belly-up.)

The final somewhat annoying bit of having a somewhat ephemeral public IP address is that you need to be careful with naked domains. I’m not a fan of naked domains per se, but at the same time I understand there is a general expectation that http://boldlyopen.com works just as well as http://www.boldlyopen.com so I will be going with the flow.

Short of the DNS powers that be finally realizing that it’s 2015 and the limitation on CNAMEs is something they should work on, the only solution is to bite the bullet and use the public IP Azure is giving you. I could have bought a few IP addresses for a very reasonable amount well within my MSDN subscription, butI care about IPv4 space so I didn’t feel my blog should squat one of the few left. Then again, while this IP may change, it is guaranteed to remain the same throughout the lifetime of the deployment and in my case I have no plans of flip-flopping this machine at all. I made a mental note to update the DNS in case I ever start from scratch and I got on with my life.

With these quirks out of the way I was finally able to wade my way into the world of FreeBSD jails: a topic for a follow-up post.

Categories: FLOSS Project Planets

Jochen Wiedmann

Tue, 2015-03-17 04:27
Honouring Terry Pratchett: Java Web Application

This Blogs name is Grumpy Apache. These days, there are excellent reasons, for being grumpy: After all, my favourite author Terry Pratchett has died, way too early. But, as Terry wrote in Going Postal about John Dearheart:
His name, however, continues to be sent in the so-called Overhead of the clacks. The full message is "GNU John Dearheart", where the G means, that the message should be passed on, the N means "Not Logged" and the U that it should be turned around at the end of the line. So as the name "John Dearheart" keeps going up and down the line, this tradition applies a kind of immortality as "a man is not dead while his name is still spoken".This means, we'll be celebrating "Being childish day" today, by adding the HTTP Header

      X-Clacks-Overhead: GNU Terry Pratchett

to our web sites. And, here's how to do that with any Java Web Application.

Its simple. First of all, you'll be adding this class to your web application. It is a so-called servlet filter. I'll quote the relevant method here:

public void doFilter(ServletRequest pReq, ServletResponse pRes,
                                         FilterChain pChain) throws IOException,
ServletException {
if (pRes instanceof HttpServletResponse) {
final HttpServletResponse res = (HttpServletResponse) pRes;
res.addHeader("X-Clacks-Overhead", "GNU-Terry-Pratchett");
}
pChain.doFilter(pReq, pRes);
}Besides, add the following snippets to your web.xml:
 
    ClacksOverheadFilter
    com.github.jochenw.clacksov.ClacksOverheadFilter
 
 
    ClacksOverheadFilter
    *
  And, that's it! No modification of servlets, or the like, just a simple addition, that you can make to any web application.
Keep in mind:
As long as we are shifting his name on the Internet, Terry isn't dead.


Categories: FLOSS Project Planets

Gary Gregory: Half off Java Persistence with Hibernate

Tue, 2015-03-17 00:51

My upcoming book is the Deal of the Day on March 19 at Manning Publications.

Half off Java Persistence with Hibernate, Second Edition. Use code dotd031915au at http://www.manning.com/bauer3/


Categories: FLOSS Project Planets

Bryan Pendleton: Thomas Ptacek on how to recruit and select programmers

Sun, 2015-03-15 17:38

Thomas Ptacek, formerly of Matasano Security, and now at Starfigher, has written a long and detailed article about how he goes about trying to recruit and select programming talent: The Hiring Post.

Ptacek's article isn't the typical four paragraph overview that you'll read in your weekly business magazine; it's detailed and born of years of pain trying to hire security software experts.

But, his essay is so good that I think it will end up on the short list of articles about the software engineering hiring process that are worth going back to over and over, like Joel Spolsky's The Guerrilla Guide to Interviewing, Michael Lopp's A Glimpse and a Hook and The Sanity Check, or Laurie Voss's This is why you never end up hiring good developers.

Let's use Ptacek's own table of contents to break it down:

  • The telling success story that motivated me to write this.

    Ptacek starts out by noting how pleased he is with a recent Matasano hire, and how he realized that they had only managed to connect because he'd changed the way their hiring process works:

    A few years ago, Matasano couldn’t have hired Alex, because we relied on interviews and resumes to hire. Then we made some changes, and became a machine that spotted and recruited people like Alex: line of business .NET developers at insurance companies who pulled Rails core CVEs out of their first hour looking at the code. Sysadmins who hardware-reversed assembly firmware for phone chipsets. Epiphany: the talent is out there, but you can’t find it on a resume.
  • The part where I join the chorus of people telling you why interviews are terrible.

    Ptacek bemoans the lack of consistency and objectivity of the interview process

    Driven in part by an oral tradition of how the last 20 years of technical job interviews has resulted in terrible hires, interviewers try to assess for “smart and gets things done”. In other words: “subjective and even more subjective x-factor”.
  • But at least read this bad thing about interviews, please.

    Ptacek makes an observation I haven't seen widely highlighted, though it should be: the hostility and stress of the technical interview is directly damaging to the potential value of the interview:

    I walked into the conference room to meet him. He appeared to be physically shaking. You didn’t need to be a psychologist to detect the nervous energy; it radiated from him, visibly, like in an R. Crumb cartoon.

    Engineering teams are not infantry squads. They aren’t selected for their ability to perform under unnatural stress. But that’s what most interview processes demand, often —– as is the case with every interview that assesses “confidence” —– explicitly so.

    As Ptacek observes, you don't want to hire people who are good at interviewing; you want to hire people who are good at writing software. So, how do you do that? Well, says Ptacek, the first thing you must do is realize you have a problem:

    For every genuinely competent and effective developer who can ace a tough dev interview, there are many more genuinely competent and effective developers who can’t. While we’re selecting for the ability to control a conversation, we’re missing ability to fix broken 2-phase commits.
  • The warm-up trick.

    Ptacek suggests that the first thing you can do is to be open about the process, and about your expectations:

    Not knowing what to expect makes candidates nervous. That’s a pointless handicap.

    At my last firm, we had the “first-call” system. Every serious applicant got, on average, 30-45 minutes of director-level time on the phone before any screening began.

  • The work-sample trick.

    Ptacek makes an ever-more-common suggestion. Instead of trying to ask people questions about programming, ask them to do some programming. But as Ptacek notes, this doesn't mean tinker-toy "reverse a string at the whiteboard" gimmicks, but rather a realistic project which you have to carefully design and provide:

    You can’t do this with a trial period where a candidate gets paid to fix whatever random bugs are in the issue-tracker. Your goal is to collect data you can use for apples-apples comparisons, which means every candidate has to be working on the same problems. You have to design a test, create a scoring rubric, and iterate.
  • One simple trick that will make your team hate you.

    In this section, Ptacek continues to hammer on the "objective, reproducible, and reliable" aspect of the hiring process:

    Interviewers hate it. Thomas’s First Law Of Interviewing: If interviewers enjoy it, you’re doing something wrong. But we kept doing it, because we found that we were comfortable with “why” we were making hire/no-hire decisions when we had facts to look at. We could look at the results of an interview and compare them to the same data generated by successful team members.
  • Grade your hiring process.

    Lastly, Ptacek reminds you not to sit back and relax, but to work to constantly improve your hiring process, just like you'd work to constantly improve any other process at your organization.

    Ask yourself some questions about your hiring process.

    Are you learning both from your successes and failures?

Ptacek really cares about this, so much so that he left Matasano to start a new company to develop his ideas into a realistic hiring tool: Announcing Starfighter

I'm not sure how Starfighter will do; for one thing, it's clearly aimed directly at the "penetration tester" sub-specialization, which is certainly an important job category, but it's just one of thousands of software specialties. I am skeptical that the CTF style of programming will be applicable to other software specialties, but these are some very smart people, and they really believe in their idea, so I'm interested to see how it all goes.

Categories: FLOSS Project Planets

Nick Kew: Verdi Requiem

Sun, 2015-03-15 17:16

A week today – Sunday March 22nd – we’re performing the Verdi Requiem at the Guildhall, Plymouth.

This is of course a big work, often described as operatic.  It is deservedly one of the most popular in the choral-orchestral repertoire, and ideally suited to a big orchestra and chorus such as the Plymouth Philharmonic.  Even the non-musical will surely have encountered highlights of it, notably the Dies Irae which is an archetype for terrifying music.  Yet despite all that it’s an easy sing, and – not least – we basses get more than our usual share of the best lines!

This is one of those concerts that is going to be tremendously exciting for performers and audience alike, and I have no hesitation recommending it to readers within evening-out distance of Plymouth.


Categories: FLOSS Project Planets

Claus Ibsen: Apache Camel Eclipse Tooling with Debugger

Sun, 2015-03-15 01:26
About 2 months ago Lars Heineman blogged about improved Apache Camel Eclipse tooling on the way as part of the JBoss Tool stack

In the upcoming release they have integrated the Camel debugger with the native Eclipse debugger, so you get the Eclipse debugging experience, when you use breakpoints, single step through the Camel routes. And you can of course see all the details of the Camel Exchange and Message on the way. And as well change the data on the fly.

A screenshot is shown below, which I have borrowed from Lars's blog.

Eclipse debugging your Apache Camel routes at runtime.Lars also worked on adding support for editing Camel endpoints uri's using a properties panel, so each option is provided individually. That work was based on Apache Camel 2.14 that has some support for this.

But since we have improved this tremendously in Apache Camel 2.15, which I recently blogged about. And therefore Lars is currently working on upgrading to Camel 2.15 so the Eclipse Tooling becomes even better.

Now imaging that we take the properties panel based on Camel 2.14 and add all the extra information and documentation we have from Camel 2.15, which will allow Eclipse to present a similar enriched properties panel as hawtio can do.

Using Camel 2.14 showing an empty properties panel (no documentation, no information about defaults, required, limited enums, etc, consumer vs producer option etc.)

Eclipse Camel properties editor based on Camel 2.14
And below the enriched Camel 2.15 which has all of above information, show currently in hawtio.

Hawtio Camel properties viewer based on Camel 2.15
So imagine that the Eclipse properties panel will be able to include out of the box:

  • documentation
  • default values
  • enum types (eg choices to select among)
  • required vs optional
  • deprecated
  • simple and java type
  • option as part of uri path or query parameter
  • consumer only option
  • producer only option
  • custom category for the option (eg security, advanced, etc.)

And with Camel 2.15, we are able to do this for all the components - they all provide all this information.

Camel 2.15 also brings to the table, that it would allow the Eclipse tooling to dynamic generate the EIP palette, as Camel include information about all the EIPs and their options as well. So imagine the Eclipse tooling is able to adjust to which version of Camel you are currently using in the project. And yeah all the EIP options is now documented as well, which the tooling can present to you.

I am really exited about the possibilities that Camel 2.15 brings to the table in terms of tooling and also runtime experience we can enhance.

We also work on JBoss Forge commands that allows to add Apache Camel to existing projects, to dockerize and/or fabric8 enable the projects - that is something for another blog. But as part of this work, we are working on commands to add/edit Camel components/endpoints. So the idea would be they can show all the endpoints uris in your project, and present a nice properties editor for you to have "type safe" editing.

Going back to the title of this blog. Yeah great work Lars and the Eclipse team, we now have a great Apache Camel debugger. And its using the same Camel debugging API that hawtio uses also - no magic trick. In fact this week I was in talk with a company that has built their data integration platform on top of Apache Camel and also leverages its debugging api, to allow its developers and users to debug the deployed Camel routes, on the platform.


Categories: FLOSS Project Planets

Uwe Schindler: The Policeman’s Horror: Default Locales, Default Charsets, and Default Timezones

Sat, 2015-03-14 17:46
Time for a tool to prevent any effects coming from them!Did you ever try to run software downloaded from the net on a computer with Turkish locale? I think most of you never did that. And if you ask Turkish IT specialists, they will tell you: “It is better to configure your computer using any other locale, but not tr_TR”. I think you have no clue what I am talking about? Maybe this article gives you a hint: “A Cellphone’s Missing Dot Kills Two People, Puts Three More in Jail”.

What you see in lots of software is a so-called case-insensitive matching of keywords like parameter names or function names. This is implemented in most cases by lowercasing or upper-casing the input text and compare it with a list of already lowercased/uppercased items. This works in most cases fine, if you are anywhere in the world, except Turkey! Because most programmers don’t care about running their software in Turkey, they do not test their software under the Turkish locale.
But what happens with the case-insensitive matching if running in Turkey? Let’s take an example:User enters “BILLY” in the search field of you application. The application then uses the approach presented before and lower-cases “BILLY” and then compares it to an internal table (e.g. our search index, parameter table, function table,...). So we search in this table for “billy”. So far so good, works perfect in USA, Germany, Kenia, almost everywhere - except Turkey. What happens in the Turkish locale when we lowercase “BILLY”? After reading the above article, you might expect it: The “BILLY”.toLowerCase() statement in Java returns “bılly” (note the dot-less i: 'ı' U+0131). You can try this out on your local machine without reconfiguring it to use the Turkish locale, just try the following Java code:assertEquals(“bılly”, “BILLY”.toLowerCase(new Locale(“tr”,“TR”)));The same happens vice versa, if you uppercase a ‘i’, it gets I with dot (‘İ’ U+0130). This is really serious, million lines of code out there in Java and other languages don’t take care that the String.toLowerCase() and String.toUpperCase() methods can optionally take a defined Locale (more about that later). Some examples from projects I am involved in:

  • Try to run an XSLT stylesheet using Apache XALAN-XSLTC (or Java 5’s internal XSLT interpreter) in the Turkish locale. It will fail with “unknown instruction”, because XALAN-XSLTC compiles the XSLT to Java Bytecode and somehow lowercases a virtual machine opcode before compiling it with BCEL (see XALANJ-2420, BCEL bug #38787).
  • The HTML SAX parser NekoHTML uses locale-less uppercasing/lowercasing to normalize charset names and element names. I opened a bug report (issue #3544334).
  • If you use PHP as your favourite scripting language, which is not case sensitive for class names and other language constructs, it will throw a compile error once you try to call a function with an “i” in it (see PHP bug #18556). Unfortunately it is unlikely that this serious bug is fixed in PHP 5.3 or 5.4!

The question is now: How to solve this?The most correct way to do this is to not lowercase at all! For comparing case insensitive, Unicode defines “case folding”, which is a so-called canonical form of text where all upper/lower case of any character is normalized away. Unfortunately this case folded text may no longer be readable text (this depends on the implementation, but in most cases it is). It just ensures, that case-folded text can be compared to each other in a case-insensitive way. Unfortunately Java does not offer you a function to get this string, but ICU-4J can do (see UCharacter#foldCase). But Java offers something much better: String.equalsIgnoreCase(String), which internally handles case folding! But in lots of cases you cannot use this fantastic method, because you want to lookup such strings in a HashMap or other dictionary. Without modifying HashMap to use equalsIgnoreCase, this would never work. So we are back at lower-casing! As mentioned before, you can pass a locale to String.toLowerCase(), so the naive approach would be to tell Java, that we are in the US or using the English language: String.toLowerCase(Locale.US) or String.toLowerCase(Locale.ENGLISH). This produces identical results but is still not consistent. What happens if the US government decides to lowercase/uppercase like in Turkey? -- OK, don’t use Locale.US (this is also too US-centric). Locale.ENGLISH is fine and very generic, but languages also change over the years (who knows?), but we want to have it language invariant! If you are using Java 6, there is a much better constant: Locale.ROOT -- You should use this constant for our lowercase example: String.toLowerCase(Locale.ROOT).
You should start now and do a global search/replace on all your Java projects (if you do not rely on language specific presentation of text)! REALLY! String.toLowerCase is not the only example of “automatic default locale usage” in the Java API. There are also things like transforming dates or numbers to strings. If you use the Formatter class, and you run it somewhere in another country, String.format(“%f”, 15.5f) may not always use a period (‘.’) as decimal separator; most Germans will know this. Passing a specific locale here helps in most cases. If you are writing a GUI in English language, pass Locale.ENGLISH everywhere, otherwise text output of numbers or dates may not match the language of your GUI! If you want Formatter to behave in a invariant way, use Locale.ROOT, too (then it will for sure format numbers with period and no comma for thousands, just like Float.toString(float) does).

A second problem affecting lot’s of software are two other system-wide configurable default settings: default charset/encoding and timezone. If you open a text file with FileReader or convert an InputStream to a Reader with InputStreamReader, Java assumes automatically, that the input is in the default platform encoding. This may be fine, if you want the text to be parsed by the defaults of the operating system -- but if you pass a text file together with your software package (maybe as resource in your JAR file) and then accidentally read it using the platform’s default charset... it’ll break your app! So my second recommendation:Always pass a character set to any method converting bytes to strings (like InputStream <=> Reader, String.getBytes(),...). If you wrote the text file and ship it together with your app, only you know its encoding!For timezones, similar examples can be found.
How this affects Apache Lucene!Apache Lucene is a full-text search engine and deals with text from different languages all the time; Apache Solr is a enterprise search server on top of Lucene and deals with input documents in lots of different charsets and languages. It is therefore essential for a search library like Lucene to be as most independent from local machine settings as possible. A library must make it explicit what input it wants to have. So we require charsets and locales in all public and private APIs (or we only take e.g. java.io.Reader instead of InputStream if we expect text coming in), so the user must take care.

Robert Muir and I reviewed the source code of Apache Lucene and Solr for the coming version 4.0 (an alpha version is already available on Lucene’s homepage, documentation is here). We did this quite often, but whenever a new piece of code is committed to the source tree, it may happen that undefined locales, charsets, or similar things appear again. In most cases it is not the fault of the committer, this happens because auto-complete in IDE automatically lists possible methods and parameters to the developer. Often you select the easiest variant (like String.toLowerCase()).

Using default locales, charsets and timezones are in my opinion a big design issue in programming languages like Java. If there are locale-sensitive methods, those methods should take a locale, if you convert a byte[] stream to a char[] stream, a charset must be given. Automatically falling back to defaults is a no-go in the server environment. If a developer is interested in using the default locale of the user’s computer, he can always explicitely give the locale or charset. In our example this would be String.toLowerCase(Locale.getDefault()). This is more verbose, but it is obvious what the developer intends to do.
My proposal is to ban all those default charset and locale methods / classes in the Java API by deprecating them as soon as possible, so users stop using them implicit!
Robert’s and my intention is to automatically fail the nightly builds (or compilation on the developer’s machine) when somebody uses one of the above methods in Lucene’s or Solr’s source code. We looked at different solutions like PMD or FindBugs, but both tools are too sloppy to handle that in a consistent way (PMD does not have any “default charset” method detection and Findbugs has only a very short list of method signatures). In addition, both PMD and FindBugs are very slow and often fail to correctly detect all problems. For Lucene builds we only need a tool, that looks into the byte code of all generated Java classes of Apache Lucene and Solr, and fails the build if any signature that violates our requirements is found.
A new Tool for the PolicemanI started to hack a tool as a custom ANT task using ASM 4.0 (Lightweight Java Bytecode Manipulation Framework). The idea was to provide a list of methods signatures, field names and plain class names that should fail the build, once bytecode accesses it in any way. A first version of this task was published in issue LUCENE-4199, later improvements was to add support for fields (LUCENE-4202) and a sophisticated signature expansion to also catch calls to subclasses of the given signatures (LUCENE-4206).

In the meantime, Robert worked on the list of “forbidden” APIs. This is what came out in the first version:java.lang.String#<init>(byte[])
java.lang.String#<init>(byte[],int)
java.lang.String#<init>(byte[],int,int)
java.lang.String#<init>(byte[],int,int,int)
java.lang.String#getBytes()
java.lang.String#getBytes(int,int,byte[],int)
java.lang.String#toLowerCase()
java.lang.String#toUpperCase()
java.lang.String#format(java.lang.String,java.lang.Object[])
java.io.FileReader
java.io.FileWriter
java.io.ByteArrayOutputStream#toString()
java.io.InputStreamReader#<init>(java.io.InputStream)
java.io.OutputStreamWriter#<init>(java.io.OutputStream)
java.io.PrintStream#<init>(java.io.File)
java.io.PrintStream#<init>(java.io.OutputStream)
java.io.PrintStream#<init>(java.io.OutputStream,boolean)
java.io.PrintStream#<init>(java.lang.String)
java.io.PrintWriter#<init>(java.io.File)
java.io.PrintWriter#<init>(java.io.OutputStream)
java.io.PrintWriter#<init>(java.io.OutputStream,boolean)
java.io.PrintWriter#<init>(java.lang.String)
java.io.PrintWriter#format(java.lang.String,java.lang.Object[])
java.io.PrintWriter#printf(java.lang.String,java.lang.Object[])
java.nio.charset.Charset#displayName()
java.text.BreakIterator#getCharacterInstance()
java.text.BreakIterator#getLineInstance()
java.text.BreakIterator#getSentenceInstance()
java.text.BreakIterator#getWordInstance()
java.text.Collator#getInstance()
java.text.DateFormat#getTimeInstance()
java.text.DateFormat#getTimeInstance(int)
java.text.DateFormat#getDateInstance()
java.text.DateFormat#getDateInstance(int)
java.text.DateFormat#getDateTimeInstance()
java.text.DateFormat#getDateTimeInstance(int,int)
java.text.DateFormat#getInstance()
java.text.DateFormatSymbols#<init>()
java.text.DateFormatSymbols#getInstance()
java.text.DecimalFormat#<init>()
java.text.DecimalFormat#<init>(java.lang.String)
java.text.DecimalFormatSymbols#<init>()
java.text.DecimalFormatSymbols#getInstance()
java.text.MessageFormat#<init>(java.lang.String)
java.text.NumberFormat#getInstance()
java.text.NumberFormat#getNumberInstance()
java.text.NumberFormat#getIntegerInstance()
java.text.NumberFormat#getCurrencyInstance()
java.text.NumberFormat#getPercentInstance()
java.text.SimpleDateFormat#<init>()
java.text.SimpleDateFormat#<init>(java.lang.String)
java.util.Calendar#<init>()
java.util.Calendar#getInstance()
java.util.Calendar#getInstance(java.util.Locale)
java.util.Calendar#getInstance(java.util.TimeZone)
java.util.Currency#getSymbol()
java.util.GregorianCalendar#<init>()
java.util.GregorianCalendar#<init>(int,int,int)
java.util.GregorianCalendar#<init>(int,int,int,int,int)
java.util.GregorianCalendar#<init>(int,int,int,int,int,int)
java.util.GregorianCalendar#<init>(java.util.Locale)
java.util.GregorianCalendar#<init>(java.util.TimeZone)
java.util.Scanner#<init>(java.io.InputStream)
java.util.Scanner#<init>(java.io.File)
java.util.Scanner#<init>(java.nio.channels.ReadableByteChannel)
java.util.Formatter#<init>()
java.util.Formatter#<init>(java.lang.Appendable)
java.util.Formatter#<init>(java.io.File)
java.util.Formatter#<init>(java.io.File,java.lang.String)
java.util.Formatter#<init>(java.io.OutputStream)
java.util.Formatter#<init>(java.io.OutputStream,java.lang.String)
java.util.Formatter#<init>(java.io.PrintStream)
java.util.Formatter#<init>(java.lang.String)
java.util.Formatter#<init>(java.lang.String,java.lang.String)
Using this easily extend-able list, saved in a text file (UTF-8 encoded!), you can invoke my new ANT task (after registering it with <taskdef/>) very easy -- taken from Lucene/Solr’s build.xml:<taskdef resource="lucene-solr.antlib.xml">
<classpath>
<pathelement location="${custom-tasks.dir}/build/classes/java" />
<fileset dir="${custom-tasks.dir}/lib" includes="asm-debug-all-4.0.jar" />
</classpath>
</taskdef>
<forbidden-apis>
<classpath refid="additional.dependencies"/>
<apiFileSet dir="${custom-tasks.dir}/forbiddenApis">
<include name="jdk.txt" />
<include name="jdk-deprecated.txt" />
<include name="commons-io.txt" />
</apiFileSet>
<fileset dir="${basedir}/build" includes="**/*.class" />
</forbidden-apis>
The classpath given is used to look up the API signatures (provided as apiFileSet). Classpath is only needed if signatures are coming from 3rd party libraries. The inner fileset should list all class files to be checked. For running the task you also need asm-all-4.0.jar available in the task’s classpath.

If you are interested, take the source code, it is open source and released as part of the tool set shipped with Apache Lucene & Solr: Source, API lists (revision number 1360240).

At the moment we are investigating other opportunities brought by that tool:
  • We want to ban System.out/err or things like horrible Eclipse-like try...catch...printStackTrace() auto-generated Exception stubs. We can just ban those fields from the java.lang.System class and of course, Throwable#printStackTrace().
  • Using optimized Lucene-provided replacements for JDK API calls. This can be enforced by failing on the JDK signatures.
  • Failing the build on deprecated calls to Java’s API. We can of course print warnings for deprecations, but failing the build is better. And: We use deprecation annotations in Lucene’s own library code, so javac-generated warnings don’t help. We can use the list of deprecated stuff from JDK Javadocs to trigger the failures.
I hope other projects take a similar approach to scan their binary/source code and free it from system dependent API calls, which are not predictable for production systems in the server environment.

Thanks to Robert Muir and Dawid Weiss for help and suggestions!

EDIT (2015-03-14): On 2013-02-04, I released the plugin as Apache Ant, Apache Maven and CLI task on Google Code; later on 2015-03-14, it was migrated to Github. The project URL is: https://github.com/policeman-tools/forbidden-apis. The tool is available to your builds using Maven/Ivy through Maven Central and Sonatype repositories. Nightly snapshot builds are done by the Policeman Jenkins Server and can be downloaded from the Sonatype Snapshot repository.
Categories: FLOSS Project Planets

Justin Mason: Links for 2015-03-13

Fri, 2015-03-13 18:58
  • demonstration of the importance of server-side request timeouts

    from MongoDB, but similar issues often apply in many other TCP/HTTP-based systems

    (tags: tcp http requests timeout mongodb reliability safety)

  • Heka

    an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing, useful for a wide variety of different tasks, such as: Loading and parsing log files from a file system. Accepting statsd type metrics data for aggregation and forwarding to upstream time series data stores such as graphite or InfluxDB. Launching external processes to gather operational data from the local system. Performing real time analysis, graphing, and anomaly detection on any data flowing through the Heka pipeline. Shipping data from one location to another via the use of an external transport (such as AMQP) or directly (via TCP). Delivering processed data to one or more persistent data stores. Via feylya on twitter. Looks potentially nifty

    (tags: heka mozilla monitoring metrics via:feylya ops statsd graphite stream-processing)

Categories: FLOSS Project Planets

Sergey Beryozkin: Talking about CXF at Apache Con NA 2015

Fri, 2015-03-13 12:31
Apache Con NA 2015 will be held in Austin, Texas on April 13-16 and as it is usually the case there will be several presentations done there about Apache CXF. There will be interesting presentations from Hadrian and JB too. There will be many other great presentations as usual.

As far as CXF presentations are concerned:

Aki Yoshida will talk about combining Swagger (Web) Sockets, Apache Olingo and CXF Web Sockets Transport - now, this is seriously cool :-) The good news the presentations will be available online for those who will not be able to see it live.

Andriy Redko will talk about something which equally rocks, about combining a CXF Search Extension (FIQL or OData/Olingo based), Apache Tika and Lucene to show the effective search support for uploaded PDF and Open Office documents.

Attending both presentations can get anyone over-excited, that is for sure :-).
This is going to be tough, choosing to which presentation to go with my other colleagues presenting on the same day.


Finally, I will do the introduction of Apache CXF JOSE implementation which I briefly introduced in the previous blog. I'll describe all that CXF JOSE project has in place, and finish with a demo.

The demo deserves a special attention: I haven't written this demo, Anders Rundgren did. The original demo is here. This appears to be like a regular JavaScript-based demo but it is bigger than that, it shows what WebCrypto can do. Supporting generic browser-based signature applications, and interoperating with target servers in a variety of formats, with JOSE one of them. So the demo will show a WebCrypto client interoperating with an Apache CXF JOSE server.


Anders has been incredibly helpful and supportive, helped me to get his demo running in no time. Anders is working on a JSON Clear Signature (JCS) initiative that offers an XML Signature like support for signing JSON documents.  JCS are easier to understand than JOSE formats where Base64URL content representations are used. I'd like to encourage the interested users experiment with JCS, and help Anders. Hopefully something similar to JCS will be supported as part of a wider JOSE effort in the future.

I'm happy as usual I've got a talk selected and my employer's support to travel to Apache Con. It is always great to talk to my colleagues who work with CXF and other Apache technologies, it is important to show others CXF is very much alive and 'looks forward'. I regret I won't see some of my team colleagues there who haven't had a chance to submit for various important reasons but overall I'm looking forward to the conference with a great anticipation. Especially because I promised someone to beat him in chess after the presentations are over :-).

See you there !






Categories: FLOSS Project Planets

Sergey Beryozkin: Apache CXF is getting JOSE ready

Fri, 2015-03-13 11:47
I've already talked about JOSE on this blog. In my opinion, it is one of the key technologies, alongside OAuth2, that will deeply affect the way developers write secure HTTP RS services in the years to come.

A one sentence summary: one can use JOSE to secure, sign and/or encrypt a data content in any format, JSON, text, binaries, anything. JOSE is a key component of an advanced OAuth2 application, but also is a good fit for securing the regular HTTP web service communications.

As such it should not be a surprise that CXF now ships its own JOSE implementation offering a support for all of JOSE signature and encryption algorithms and representation formats and joins a list of other frameworks/projects directly supporting JOSE.

I've done an initial documentation here. There's so much to document that I will need probably another week to complete it all. Lots of interesting stuff for developers to experiment with that needs to be documented. I think it is unique in its own way while probably repeating some of the boilerplate code that any JOSE implementation needs to do.

Apart from being keen to directly deal with such an implementation, IMHO it is also good to have it supported in CXF due to how important this technology will become for web services developers in the future. It is always healthy to have multiple implementations as the JAX-RS space has demonstrated. And if CXF users would prefer to use other JOSE implementations then it will be fine.

One such 3rd party implementation is Jose4J. I'd like to thank Brian Campbell for creating it - it did help me to keep my sanity when I only started trying to write a test validating an RSA-OAEP output which is random. I also looked at its source recently when I was puzzled as to why my tests involving EC keys produce wrong-size signatures, even though the validation was passing - the comment in Jose4J made a rather cryptic JOSE spec text obvious, JOSE EC signatures are formatted in a format more compact than DER. I still wrote my own code though :-) which one might say is questionable but there you go. Thanks Brian. I think we can plug in Jose4J with CXF JOSE filters easily enough should users demand it.



CXF JOSE project is not completely finalized but I'm thinking it is getting pretty close to the final API. I'd like to encourage the early adopters give it a go and provide the feedback. In meantime I'll be working on completing the documentation and tweaking the code to enforce some of the security considerations documented in JOSE specifications, etc.

Enjoy !




Categories: FLOSS Project Planets

Bryan Pendleton: What I'm reading, mid-March edition

Fri, 2015-03-13 10:40

The fruit trees are all happy, everything is in bloom.

And I'm reading a completely random collection of stuff...

  • Disambiguating DatabasesThe scope of the term database is vast. Technically speaking, anything that stores data for later retrieval is a database. Even by that broad definition, there is functionality that is common to most databases. This article enumerates those features at a high level. The intent is to provide readers with a toolset with which they might evaluate databases on their relative merits.
  • There is No NowOne of the most common mistakes in building systems—especially, but not only, distributed-computing systems on commodity machines and networks—is assuming an escape from basic physical realities. The speed of light is one such reality, but so is one that is more pernicious and just as universal: we cannot make perfect machines that never break. It is the combination of these realities, of asynchrony and partial failure, that together make building distributed systems a difficult pursuit. If we do not plan and account for failures in individual components, we all but guarantee the failure of combined systems.
  • The future of Team Foundation Version controlWe've been investing heavily in Git because there's a ton of work to do to bring it up to parity with what we can do with TFVC. I think people get confused for a number or reasons. We talk about our progress on Git a lot. The industry talks about Git a lot. And, if you are watching, you'll hear more and more about teams inside Microsoft adopting Git. My own team has moved a bunch of stuff to Git.
  • How GitHub Conquered Google, Microsoft, and Everyone ElseGoogle Code is dying because most of the open source world—a vast swath of the tech world in general—now houses its code on GitHub, a site bootstrapped by a quirky San Francisco startup of the same name. All but a few of those thousand projects are now on GitHub.
  • Open-sourcing PinballAfter experimenting with a few open-source workflow managers we found none of them to be flexible enough to accommodate the ever-changing landscape of our data processing solutions. In particular, current available solutions are either scoped to support a specific type of job (e.g. Apache Oozie optimized for Hadoop computations) or abstractly broad and hard to extend (e.g. monolithic Azkaban). With that in mind, we took on the challenge of implementing a highly customizable workflow manager build to survive the evolution of the data processing use cases ranging from execution of basic shell commands to elaborate ETL-style computations on top of Hadoop, Hive and Spark.
  • The sad state of sysadmin in the age of containersNobody seems to know how to build Hadoop from scratch. It's an incredible mess of dependencies, version requirements and build tools.

    None of these "fancy" tools still builds by a traditional make command. Every tool has to come up with their own, incomptaible, and non-portable "method of the day" of building.

    And since nobody is still able to compile things from scratch, everybody just downloads precompiled binaries from random websites.

  • Booster TestNASA socials give some inside access to people like me (and you, if you live in the USA and want to sign up next time) who have no official connection to the space program. Yesterday we got to tour the plant where the boosters are made. It was great to learn about techniques for mixing, casting, and curing huge amounts of propellant without getting air bubbles or other imperfections into the mix and without endangering workers. The buildings in this part of ATK have escape slides from all levels and are surrounded by big earthworks to deflect potential explosions upwards. It was also really cool to see the hardware for hooking boosters to the main rocket, for vectoring nozzles, and things like that. Alas, we weren’t allowed to take pictures on the tour.
  • Wild-Winter Whodunnit—Climate Change Over the U.S. With a Slow Jet Stream?It’s a notion so vast in scale it’s difficult to imagine, yet the mechanics are fairly simple. The jet stream is generated by a combination of Earth’s rotation and the flow of air down atmospheric gradients between high-pressure, mid-latitude warmth and low-pressure Arctic cold. Over the last several decades, the Arctic has warmed faster than any other region; during periods of especially heightened warming, as occurs when melting sea ice exposes dark, sunlight-absorbing waters, the north-to-south temperature difference shrinks. The pressure differences flatten.

    This decreased gradient slows down the jet stream—and as it slows, it also seems to become wavier, plunging south or veering north when encountering atmospheric obstacles it would once have coursed straight through.

  • The 2015 California Earthquake ForecastWe can’t deal with the situation using simple, linear computer models based on one idea of Earth’s behavior. The third UCERF is a supple, fine-grained instrument that takes advantage of many significant advances during the last decade. When I told a USGS quake guy yesterday how much I admired the new model, his eyes twinkled. They’re proud of this.
  • Great ladies of history find a new home in strategy gamesParadox's chief operating officer is Susana Meza-Graham, and she says she loves working in the demanding strategy game space. "I love that the games we develop and publish within the strategy space matter for others," she says. "We have countless examples of teachers using our games as a way to bring history, politics and economics to life for their students."

    She shows me a TED talk from Alex Petroff, who at 6:38 minutes in says Paradox games have helped him run an NGO dedicated to getting rural people out of poverty in the Congo.

    "We've never gone out of our way to label a game as typically 'male' or 'female'," Meza-Graham says. "Instead we try to find appealing ways to communicate the merits of the game and what makes them special and worth playing."

    Crusader Kings II in particular has a higher ratio of female players than many of the company's other strategy games. Meza-Graham puts the total at about 40 percent.

Categories: FLOSS Project Planets

Matt Raible: Skiing Adventures with our Syncro in the Rocky Mountains

Fri, 2015-03-13 07:43

Our Syncro Westy was returned to us in early February, after being in the shop since just before Christmas. It was in the shop for body work caused by an accident that was my fault. Luckily, no one was hurt and the damage was minor. The morning after we got it back, my awesome friend Ryan Moore and I packed it up and headed on a hut trip near Aspen, Colorado.

The next weekend was one of the most popular skiing holidays: Presidents' Day Weekend. We packed up the kids and drove our Ski Bus to Crested Butte. I took my guitar and Trish took her banjo (we both started taking weekly lessons at the beginning of the year). Our drive was smooth and our weekend was fabulous.

Saturday was Valentine's Day and I surprised Trish with a photo shoot of our family. I'd secretly hired Alison White to take our pictures and we met with her to talk about what we wanted on Saturday morning. After a fun consultation, we ventured to the mountain, picking up James Ward along the way. We skied a few runs together, stopped at the Ice Bar for a car bomb, then whisked off to our photo shoot.

We are extremely pleased with the results. Thanks Alison!

The next weekend, Trish and I packed up and headed to our Ski Shack in Winter Park. We stayed up late that Friday night, preparing chili and other fixins for a fun tailgate on Saturday. We didn't get much sleep, but we were in "C Lot" by 7:30 and on the lifts shortly after they opened. It was a powder day, and the runs were great. We met up with friends at our van around noon and enjoyed some tasty chilly and cold beverages.

The last weekend in February, we met Trish's brother (also named Matt) in Vail. He was on a ski trip with a number of friends and Trish drove up to ski with him on Saturday morning. The kids and I slept in a bit and arrived at Copper just in time for free parking near the lifts. We skied all afternoon, then met up with Trish and Matt for some cocktails, then dinner. That night, we did some winter camping in the van.

The final weekend in my 5-weekends-in-a-row of skiing was a wonderful trip to Telluride. The kids had Thursday and Friday off of school last week, so we loaded up and drove to Telluride on Thursday. It took us most of the day to get there (7.5 hours), but the weather was beautiful and the roads were dry. Our van was too tall for our hotel's garage, so we had excellent parking spots all weekend.

This was the breakout weekend for Jack. Abbie's been skiing awesome all year and Jack's been pretty timid (lots of turns, going slow). There was something about our first day at Telluride that inspired him, because he loved their double blacks. We skied lots of steep runs, did laps on the terrain parks and flew down the mountain on blues. Telluride had a Spring Break atmosphere and our ski-in/ski-out accommodations didn't hurt. It was a glorious weekend. Trish and I were both very proud of the kids and how fast and fearless they've become on skis.

Full Album →

I'm extremely happy with our Syncro's performance and its new H6 (Subaru Outback) engine. It's been averaging 20mpg, most likely because I drive 55mph in the slow lanes up the passes. I figure we drove it just over 2000 miles on our 5-ski-weekends-in-5-weeks adventure. We thoroughly enjoyed skiing at Crested Butte, tailgating with powder at Mary Jane and shredding the double blacks at Telluride. I know we'll be back to all of them, some sooner than later.

In fact, I'll be meeting a few friends at Mary Jane this morning to attempt 30,000 vertical feet in one day. It'll be my 36th day of skiing this year. Happy Friday! Life is Good.

Categories: FLOSS Project Planets

Shawn McKinney: Enabling Java EE and Fortress Security inside an Apache Wicket Web App

Thu, 2015-03-12 23:22

    

Introduction

The aim of this tutorial is to demonstrate how a simple wicket web app can be converted to use java EE and fortress security.  It’s not intended to highlight all of the possible locations in code where security checks may be applied.  For that take a look at The Apache Fortress End-to-End Security Tutorial.

Course Prerequisites

Before starting, you need a working fortress environment up and running.  Complete the following steps under the Apache Fortress Ten Minute Guide:

 Steps to Enable Security in Wicket Sample 1. Download the fortress wicket-sample project a. extract the zip b. follow the instructions in README.md

You may now hit the sample web app:

http://localhost:8080/wicket-sample

no security

This is the view you will see when clicking on link for Page1.  Despite what it says about rbac perms, it’s an example of an app that’s not secured.  Users are not required to logon and may navigate freely.

To enable security for this app, continue on…

The following instructions are duplicated by: README-ENABLE-FORTRESS.md

2. Add the Java EE security required artifacts

If you are using the wicket-sample source, this is already done.  It includes wicket components LoginPage.java, LogoutPage.java (plus associated markup files LoginPage.html, LogoutPage.html), and the static html files under the /login folder.  These files control the flow between the container and wicket with java EE security enabled.

java EE login page

more on java EE security

Declarative style of security, one with no api calls made by programmer (during logon).  This makes java EE security both error resistant and hardened to attack, though it tends to get in the way of the page flows.  We’ll cover this topic, in more detail, on a future post.

3. Edit the pom.xml

Prepare maven for fortress.

a. Uncomment the fortress web dependency at the top of the file. <!-- TODO STEP 3: uncomment for fortress security dependency: --> <dependency> <groupId>org.apache.directory</groupId> <artifactId>fortress-web</artifactId> <version>${project.version}</version> <classifier>classes</classifier> </dependency> b. Uncomment the maven ant task near the bottom of the file.

This plugin calls the fortress load procedure during normal maven tasks like install or deploy.  The fortress load inserts the app’s security policy into ldap.  It can be overridden by passing param -Dnoload which tells the plugin not to execute.

<plugin> <artifactId>maven-antrun-plugin</artifactId> <version>1.7</version> <executions> <execution> <phase>test</phase> <configuration> <target name="fortress-load" if="load.file" description="Loads wicket sample security policy into ldap"> ...

At the completion of this step, the necessary binaries will be available to the app and the app’s security policy file will be ready to load.

4. Edit the web.xml

Prepare the app for fortress.

a. Uncomment the spring settings. <!-- TODO STEP 4a: uncomment to enable fortress spring bean injection: --> <context-param> <param-name>contextConfigLocation</param-name> <param-value>classpath:applicationContext.xml</param-value> </context-param> <listener> <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class> </listener>

Notice that we’re including a link to a spring context file: applicationContext.xml:

<?xml version="1.0" encoding="UTF-8"?> ... <bean id="accessMgr" class="org.apache.directory.fortress.core.AccessMgrFactory" scope="prototype" factory-method="createInstance"> <constructor-arg value="HOME"/> </bean> <bean id="j2eePolicyMgr" class="org.apache.directory.fortress.realm.J2eePolicyMgrFactory" scope="prototype" factory-method="createInstance"> </bean> ...

The context file holds the metadata necessary to wire the fortress objects in with their constructors and subsequently get injected into the web app as spring beans.

b. Uncomment the java ee security constraints from web.xml: <!-- TODO STEP 4b: uncomment to enable Java EE Security: --> <security-constraint> ...

Now container security has been enabled for this web app.  It authenticates, checks roles and maintains the session.

The wicket logout page is excluded from java EE security constraints.  Otherwise, a problem occurs preventing unauthorized users from logging out.

<security-constraint> <web-resource-collection> <web-resource-name>app</web-resource-name> <url-pattern>/wicket/bookmarkable/org.wicketsample.LogoutPage</url-pattern> </web-resource-collection> <!-- OMIT auth-constraint --> </security-constraint> 5. Rename context.xml.example to context.xml

Prepare the app for the fortress realm.

<Context reloadable="true"> <Realm className="org.apache.directory.fortress.realm.tomcat.Tc7AccessMgrProxy" debug="0" resourceName="UserDatabase" defaultRoles="" containerType="TomcatContext" realmClasspath="" /> </Context>

This file hooks a web app into the tomcat fortress realm which performs security functions like authenticate and isUserInRole. It’s also where the security session gets created by fortress.

more on the realm

The fortress realm’s proxy jar must be present under tomcat’s lib folder (as discussed in the fortress ten minute guide).

The proxy jar is a shim that uses a URLClassLoader to reach its implementation libs.  The proxy prevents the realm impl libs, pulled in as dependency to web app, from interfering with the container’s system classpath thus providing an error free deployment process free from classloader issues.  The proxy offers the flexibility for each web app to determine its own version/type of security realm to use, satisfying a variety of requirements related to web hosting and multitenancy.

6. Rename fortress.properties.example to fortress.properties.

Prepare fortress for ldap server usage.

After completing the fortress ten minute guide, this step should be familiar to you.  It is how the fortress runtime gets hooked in with a remote ldap server.

# This param tells fortress what type of ldap server in use: ldap.server.type=apacheds # ldap host: host=localhost # ldap port: port=10389 # These credentials are used for read/write access to all nodes under suffix: admin.user=uid=admin,ou=system admin.pw=secret # This is min/max settings for LDAP administrator pool connections that have read/write access to all nodes under suffix: min.admin.conn=1 max.admin.conn=10 # This node contains fortress properties stored on behalf of connecting LDAP clients: config.realm=DEFAULT config.root=ou=Config,dc=example,dc=com # Used by application security components: perms.cached=true # Fortress uses a cache: ehcache.config.file=ehcache.xml 7. Edit WicketApplication.java

Tell wicket about fortress sessions and objects.

a. Uncomment fortress session override.

Here we override app’s wicket session with a new one that can hold onto fortress session and perms:

// TODO STEP 7a: uncomment save fortress session to wicket session: @Override public Session newSession(Request request, Response response) { return new WicketSession(request); } b. Uncomment fortress spring bean injector.

Next we tell the app to use spring to inject references to fortress security objects:

// TODO STEP 7b: uncomment to enable injection of fortress spring beans: getComponentInstantiationListeners().add(new SpringComponentInjector(this));

These steps are necessary to get fortress runtime wired into the sample app.

8. Edit WicketSampleBasePage.java

Get fortress objects injected to the wicket base page, enable fortress secured page links.

a. Uncomment fortress spring bean injection.

This step automatically instantiates the fortress objects needed for security processing.

// TODO STEP 8a: enable spring injection of fortress bean here: @SpringBean private AccessMgr accessMgr; @SpringBean private J2eePolicyMgr j2eePolicyMgr;

These objects are used by the app to make AccessMgr calls to functions like checkAccess and sessionPermissions.

b. Uncomment call to enableFortress.

This performs the boilerplate security functions required by fortress during app session startup:

// TODO STEP 8b: uncomment call to enableFortress: try { SecUtils.enableFortress( this, ( HttpServletRequest ) getRequest().getContainerRequest(), j2eePolicyMgr, accessMgr ); } catch (org.apache.directory.fortress.core.SecurityException se) { ... } c. Change to FtBookmarkablePageLink

The advantage here is other than a name change, everything else stays the same, and now the links are secured.

// TODO STEP 8c: change to FtBookmarkablePageLink: add( new FtBookmarkablePageLink( "page1.link", Page1.class ) ); add( new FtBookmarkablePageLink( "page2.link", Page2.class ) ); add( new FtBookmarkablePageLink( "page3.link", Page3.class ) );

This component maps a page link to a fortress permission.  The wicket id passed in, e.g. page1.link, is converted to a fortress permission, objName: page1, opName: link.

9. Edit Page1.java, Page2.java, Page3.java

Enable fortress secured buttons.  Each page has three buttons.  Same as before, only the name changes.

a – i. change the button class name to FtIndicatingAjaxButton: // TODO STEP 9a: change to FtIndicatingAjaxButton: add( new FtIndicatingAjaxButton( "page1.button1" )

This component maps the buttons to fortress permissions.  The wicket id, e.g. page1.button1, is converted to a fortress permission, objName: page1, opName: button1.

10. Build & Deploy Web App

Deploy to tomcat server:

mvn clean tomcat:deploy -Dload.file

Or if already deployed:

mvn clean tomcat:redeploy -Dload.file

The -Dload.file system parameter tells maven to load the wicket sample security policy into ldap in addition to deploying the app.  Since the load needs to happen just once, you may drop it from future invocations:

mvn tomcat:redeploy

We’re done with setup.  Now onto testing…

Testing the Sample App Security Policy

The wicket-sample security policy is loaded automatically during maven deployment step.  The file:  wicket-sample-security-policy.xml

How to create an rbac policy was covered recently:

The Seven Steps of Role Engineering

The diagram below depicts the wicket sample role hierarchy and its user to role assignments:

sample security policy

  • wsBaseRole is inherited by all other roles
  • wsSuperUsers inherits wsUsers1Role, wsUsers2Role and wsUsers3Role
  • wsUser1 is assigned wsUsers1Role and has access to buttons on Page1
  • wsUser2 is assigned wsUsers2Role and can hit buttons on Page2
  • wsUser3 is assigned wsUsers3Role and has buttons on Page3
  • wsSuperUser is assigned wsSuperUser role has all pages and all buttons
Use Cases for Testing a. wssuperuser/password

Receives links for Page1, Page2 and Page3.

wsSuperUser launch page

b. wsuser1/password

Receives a page link for Page1.  May only view and click on Page1’s buttons.

wsUser1 launch page

c. wsuser2/password

Receives link for Page2.  May only view and click on Page2’s buttons.

wsUser2 launch page

d. wsuser3/password

Receives link for Page3.  May only view and click on Page3’s buttons.

wsUser3 launch page

More on sample security policy

As stated in the beginning, the sample wicket security controls are not comprehensive.  For example, a user may navigate by manually entering page urls thus sidestepping the secured page link component.  In this scenario the user will be prevented from clicking on buttons unless otherwise authorized.  Still, allowing users to navigate to unauthorized pages is a bad practice.  The Apache Fortress End-to-End Security Tutorial uses spring security controls to lock down pages.  There are many other ways – how would you do it?


Categories: FLOSS Project Planets

Justin Mason: Links for 2015-03-12

Thu, 2015-03-12 18:58
Categories: FLOSS Project Planets

Luciano Resende: Disable "Remove trailing whitespace" defaults in Atom editor

Thu, 2015-03-12 10:51
If you are using Atom.io text editor, you might have noticed that it automatically removes all the trailing white space on your code. This can be very annoying when you are working on changes, and the formatting changes are getting mixed with logic changes. To disable this, update the config.cson file in your home directory :
cd ~/.atom
vi config.cson

...
'global':
'whitespace':
'removeTrailingWhitespace': false

...

After restarting your editor app, you should be all set.
Categories: FLOSS Project Planets

Justin Mason: Links for 2015-03-11

Wed, 2015-03-11 18:58
Categories: FLOSS Project Planets

Adrian Sutton: Patterns are for People

Wed, 2015-03-11 17:01
Avdi Grimm in Patterns are for People:Patterns aren’t tools for programming computers; they are tools for programming people. As such, to say that “patterns are a language smell” makes very little sense. Patterns are a tool for augmenting language: our language, the language we use to talk to each other about a problem and its solutions; the language we use to daydream new machines in our minds before committing them to code. Patterns aren’t a language smell; rather, patterns are by definition, [optional] language features. There’s so much we can learn if only we stop making everything into a battle between “new” tools or approaches and “old” ones. Patterns are relevant and useful in all forms of programming and we shouldn’t discard them just because they’re not “cool” anymore. Sadly we seem to throw out nearly all our experience and learning each time a new approach comes along rather than learning from and taking advantage of both.
Categories: FLOSS Project Planets

Sergey Beryozkin: Camel CXFRS Improvements

Wed, 2015-03-11 12:51
Camel CXFRS is one of the oldest Camel components which was created by Willem Jiang, my former colleague back from IONA Technology days, and maintained by Willem since its early days.

Camel is known to be a very democratic project with respect to supporting all sort of components, and it has many components that can deal with HTTP invocations. CXFRS is indeed just one of them but as you can guess from its name it is dedicated to supporting HTTP endpoints and clients written on top of Apache CXF JAX-RS implementation.

I think that over the years CXFRS has got a bit of the mixed reception from the community,  may be because it was not deemed that ideal for supporting some styles of routing for which other lighter Camel HTTP aware components were good at.

However CXFRS has been used by some developers and it has been significantly improved recently with respect to its usability. I'd like though to touch on the very last few updates which can be of interest.

The main CXFRS feature which appears to be quite confusing initially is that a CXFRS endpoint (Camel Consumer)  does not actually invoke on the provided JAX-RS implementation. This appears to be rather strange but this is what actually helps to integrate CXF JAXRS into Camel. The JAX-RS runtime is only used to prepare all the data according to JAX-RS Service method signatures but not invoke the actual service but make all the data needed available to custom Camel processors which extract these data from Camel exchanges and make some next routing decisions.

The side-effect of it that in some cases once can not actually just take an existing JAX-RS service implementation and plug it into a Camel route. Unless one use a CXFRS Bean component that can route from Jetty endpoints to CXF JAX-RS service implementation. This approach works but requires another Camel (Jetty only) component with an absolute HTTP address and has a few limitations of its own.

So the first improvement is that starting from Camel 2.15.0 one can configure a CXFRS consumer with a 'performInvocation=true' option and it will actually invoke on the service implementation, set a JAX-RS response on the Camel  exchange and will route to the next custom processor as usual, except that in this case the custom processor will have all the input parameters as before but also a response ready - the processors now can customize the response or do whatever else they need to do. It also makes it much simpler to convert the existing CXF Spring/Blueprint JAX-RS declarations  with the service implementations into Camel CXFRS endpoints if needed.

Note that in a default case one typically provides a no-op CXFRS service implementation (recall, CXFRS does not invoke on the service by default, only needs the method signatures/JAX-RS metadata). Providing interfaces only makes it more logical given that the invocation is not done by default, in fact it is possible for URI-only CXFRS consumer style which is rather limited in what it can do. So the other minor improvement is that starting from Camel 2.15.0 one can just prepare a JAX-RS interface and use it with CXFRS Consumer unless a new 'performInvocation' option is set in which case a complete implementation is needed.

The next one is the new "propagateContexts" configuration option. What it does is that it allows CXFRS developers write their custom processors against JAX-RS Context API, i.e, they can extract one of JAX-RS Contexts such as UriInfo, SecurityContext, HttpHeaders as a typed Camel exchange property and work with these contexts to figure out what needs to be done next. This should be a useful option indeed as JAX-RS Context API is very useful indeed.

Finally, a CXF No Annotations Feature is now supported too, CXFRS users can link to a CXF Model document and use it to JAX-RS enable a given Java interface without JAX-RS annotations. In fact, starting from Camel 2.15.0 it is sufficient to have a model-only CXFRS Consumer without a specific JAX-RS service interface or implementation - in this case custom processors will get the same request data as usual, with the model serving as the source binding the request URI to a set of request parameters.

We hope to build upon this latest feature going forward with other descriptions supported, to have a model-only CXFRS consumer more capable.

Enjoy !







Categories: FLOSS Project Planets