• Excel, untestability, and the reliability of quants

  Wow, this is a great software-quality story — I knew Excel was the most widely used programming environment out there, but this is a factor I’d overlooked:

  In his remarks on the final panel, Frank Partnoy mentioned something I missed when it came out a few weeks ago: the role of Microsoft Excel in the “London Whale” trading debacle. [..] To summarize: JPMorgan’s Chief Investment Office needed a new value-at-risk (VaR) model for the synthetic credit portfolio (the one that blew up) and assigned a quantitative whiz [...] to create it. The new model “operated through a series of Excel spreadsheets, which had to be completed manually, by a process of copying and pasting data from one spreadsheet to another.” The internal Model Review Group identified this problem as well as a few others, but approved the model, while saying that it should be automated and another significant flaw should be fixed. After the London Whale trade blew up, the Model Review Group discovered that the model had not been automated and found several other errors. Most spectacularly, “After subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended. This error likely had the effect of muting volatility by a factor of two and of lowering the VaR …” I write periodically about the perils of bad software in the business world in general and the financial industry in particular, by which I usually mean back-end enterprise software that is poorly designed, insufficiently tested, and dangerously error-prone. But this is something different. [...] While Excel the program is reasonably robust, the spreadsheets that people create with Excel are incredibly fragile. There is no way to trace where your data come from, there’s no audit trail (so you can overtype numbers and not know it), and there’s no easy way to test spreadsheets, for starters. The biggest problem is that anyone can create Excel spreadsheets — badly. Because it’s so easy to use, the creation of even important spreadsheets is not restricted to people who understand programming and do it in a methodical, well-documented way. This is why the JPMorgan VaR model is the rule, not the exception: manual data entry, manual copy-and-paste, and formula errors. This is another important reason why you should pause whenever you hear that banks’ quantitative experts are smarter than Einstein, or that sophisticated risk management technology can protect banks from blowing up. At the end of the day, it’s all software. While all software breaks occasionally, Excel spreadsheets break all the time. But they don’t tell you when they break: they just give you the wrong number.

  (tags: excel reliability software coding ides jpmorgan value-at-risk finance london-whale quants spreadsheets unit-tests testability testing)

• Riak, CAP, and eventual consistency

  Good (albeit draft) write-up of the implications of CAP, allow_mult, and last_write_wins conflict-resolution policies in Riak:

  As Brewer’s CAP theorem established, distributed systems have to make hard choices. Network partition is inevitable. Hardware failure is inevitable. When a partition occurs, a well-behaved system must choose its behavior from a spectrum of options ranging from “stop accepting any writes until the outage is resolved” (thus maintaining absolute consistency) to “allow any writes and worry about consistency later” (to maximize availability). Riak leans toward the availability end of the spectrum, but allows the operator and even the developer to tune read and write requests to better meet the business needs for any given set of data.

  (tags: riak cap eventual-consistency distcomp distributed-systems partition last-write-wins voldemort allow_mult)

• How You Can Help Save Upcoming.org, Posterous, and More

  Yahoo! sucks. shutting down in days? ArchiveTeam Warrior to the rescue; install the VM!

  (tags: archival yahoo shutdowns upcoming waxy archives virtualbox)

• The Excel Depression – NYTimes.com

  Krugman on the Reinhart-Rogoff Excel-bug fiasco.

  What the Reinhart-Rogoff affair shows is the extent to which austerity has been sold on false pretenses. For three years, the turn to austerity has been presented not as a choice but as a necessity. Economic research, austerity advocates insisted, showed that terrible things happen once debt exceeds 90 percent of G.D.P. But “economic research” showed no such thing; a couple of economists made that assertion, while many others disagreed. Policy makers abandoned the unemployed and turned to austerity because they wanted to, not because they had to. So will toppling Reinhart-Rogoff from its pedestal change anything? I’d like to think so. But I predict that the usual suspects will just find another dubious piece of economic analysis to canonize, and the depression will go on and on.

  (tags: paul-krugman economics excel coding bugs software austerity debt)

• Vaccination ‘herd immunity’ demonstration

  ‘Stochastic monte-carlo epidemic SIR model to reveal herd immunity’. Fantastic demo of this important medical concept (via Colin Whittaker)

  (tags: via:colinwh stochastic herd-immunity random sir epidemics health immunity vaccination measles medicine monte-carlo-simulations simulations)

• Fred’s ImageMagick Scripts: SIMILAR

  compute an image-similarity metric, to discover mostly-identical-but-slightly-tweaked images:

  SIMILAR computes the normalized cross correlation similarity metric between two equal dimensioned images. The normalized cross correlation metric measures how similar two images are, not how different they are. The range of ncc metric values is between 0 (dissimilar) and 1 (similar). If mode=g, then the two images will be converted to grayscale. If mode=rgb, then the two images first will be converted to colorspace=rgb. Next, the ncc similarity metric will be computed for each channel. Finally, they will be combined into an rms value. (via Dan O’Neill)

  (tags: image photos pictures similar imagemagick via:dano metrics similarity)

• A Slower Speed of Light

  a first-person game prototype in which players navigate a 3D space while picking up orbs that reduce the speed of light in increments. Custom-built, open-source relativistic graphics code allows the speed of light in the game to approach the player’s own maximum walking speed. Visual effects of special relativity gradually become apparent to the player, increasing the challenge of gameplay. These effects, rendered in realtime to vertex accuracy, include the Doppler effect (red- and blue-shifting of visible light, and the shifting of infrared and ultraviolet light into the visible spectrum); the searchlight effect (increased brightness in the direction of travel); time dilation (differences in the perceived passage of time from the player and the outside world); Lorentz transformation (warping of space at near-light speeds); and the runtime effect (the ability to see objects as they were in the past, due to the travel time of light). Players can choose to share their mastery and experience of the game through Twitter. A Slower Speed of Light combines accessible gameplay and a fantasy setting with theoretical and computational physics research to deliver an engaging and pedagogically rich experience.

  (tags: games physics mit science light relativity)

• Eventual Consistency Today: Limitations, Extensions, and Beyond – ACM Queue

  Good overview of the current state of eventually-consistent data store research, covering CALM and CRDTs, from Peter Bailis and Ali Ghodsi

  (tags: eventual-consistency data storage horizontal-scaling research distcomp distributed-systems via:martin-thompson crdts calm acid cap)

• Latency’s Worst Nightmare: Performance Tuning Tips and Tricks [slides]

  the basics of running a service stack (web, app servers, data stores) on AWS. some good benchmark figures in the final slides

  (tags: benchmarks aws ec2 ebs piops services scaling scalability presentations)

• Rob “b3ta” Manuel in Dublin next week

  The Bottom Half Of The Internet — “Racism; typos; filth; spam; ignorance; rage – that’s all the bottom half of the internet is good for, right? Rob Manuel wants you to question the internet dictum, most beloved of high-profile columnists, that you should ignore all of the comments all of the time. The ‘war on comments’, he reckons, might just be an echo of a fourth estate that’s having trouble adjusting to the idea of an unwashed public disagreeing with their sacred opinions. Sous les pavés, la plage.” On Tuesday, le cool Dublin & Pilcrow present SPIEL. Rob Manuel is the flashy animator behind B3ta and he’s joined by Ed Melvin, who wants to educate you on ‘The Unreal Engines’ of virtual currencies and economies.

  (tags: rob-manuel b3ta dublin comments internet meetings talks lecool)

... and an old man's thoughts turn to Abe Maslow's (corrected) hierarchy of needs...

• Essentials of Garbage CollectionHow do we teach students the essential ideas behind garbage collection?

  A garbage collector (GC) implementation must address many overlapping concerns. There are algorithmic decisions, e.g., the GC algorithm to use; there are representation choices, e.g, the layout of the stack; and there are assumptions about the language, e.g., unforgeable pointers. These choices, and more, affect each other and make it very difficult for students to get off the ground.

• Stanford NetSeminar - Teemu Koponen (Nicira/VMware)
• Privacy technologies: An annotated syllabusLast semester I taught a course on privacy technologies here at Princeton. Earlier I discussed how I refuted privacy myths that students brought into class. In this post I’d like to discuss the contents of the course.
• Jeremy Cole: InnoDBI’ve written the following posts about InnoDB internals, structures, and behavior I particularly liked B+Tree index structures in InnoDBHowever, the root page itself can't actually be split, since it cannot be relocated. Instead, a new, empty page is allocated, the records in the root are moved there (the root is "raised" a level), and that new page is split into two. The root page then does not need to be split again until the level immediately below it has enough pages that the root becomes full of child page pointers (called "node pointers"), which in practice often means several hundred to more than a thousand.
• The Matasano Crypto ChallengesMost of the challenges take the form of practical attacks against common vulnerabilities, many of which will be sadly familiar to you from your own web apps. To keep things fun and fair for everyone, they ask you not to post the questions or answers online. (I cleared this post with Thomas to make sure it was spoiler-free.)
• The Planetary Super-Surface of San Bernardino CountyAs it happens, though, huge volumes of empty space framed by walls and ceilings are something of the ultimate testing ground for robot intelligence: "Once upon a time, a warehouse was where you stored things for weeks or months, such as toys and canned food that retailers would grab to restock their shelves. Sorting, organizing and moving the inventory was a constant challenge."

  However, now, in this age of empty architectural airspace, "Tracking goods in the modern age of bar codes, scanners and computers is a comparative breeze. The location of every widget can be identified with pinpoint accuracy and fetched by robots that can lift and carry 3,000-pound loads with ease."

• Text of SXSW2013 closing remarks by Bruce SterlingAnd yes — the moral here is that you’re a lot like that. Only they managed to pull that off for one hundred and fifty years, while you’re only twenty-six years old. If I was going to compare you to the Sinagua people of the Southwest, we’d have to imagine this as South By South West 150.
• What are Ethernet, IP and TCP Headers in Wireshark Captures and How Headers Encapsulate in the OSI stackEach device along the path from source to destination (user to Youtube in this case) strips the header it is concerned with and imposes the new information in another header wrapping the packet back up for its next hop. The payload remains the same along the way outside of fragmentation but these days Path MTU Discovery (PMTUD) has alleviated the need to fragment packets thus reducing overhead on devices in the path.
• The Well Deserved Fortune of Satoshi Nakamoto, Bitcoin creator, Visionary and GeniusThe graphs were made by a new block chain analysis technique I tested that consist on tracking the ExtraNonce fields in the coinbase field of the coinbase transaction, which is the one that creates bitcoins.
• A Practical Intro to Data ScienceThere are plenty of articles and discussions on the web about what data science is, what qualities define a data scientist, how to nurture them, and how you should position yourself to be a competitive applicant. There are far fewer resources out there about the steps to take in order to obtain the skills necessary to practice this elusive discipline.
• Lastly, my former colleague Dave Kellogg is blogging again (Hurray!)
  • Startup CEOs and the Three DoorsI think most people fail to grasp the commitment that hired (non-founder) CEOs make when joining a startup. First, the new CEO is are typically leaving a perfectly reasonable job to join the startup, so there can be a significant opportunity cost. Second, and more importantly, the new CEO is voluntarily signing up for a situation from which there are only three exit doors — because from the board’s perspective (and the venture capitalists on it) there are really only three possible outcomes
  • The Self-Fulfilling 3x Pipeline Coverage ProphecyWhat management should do is to beat on salesreps to show the real pipeline, as they believe it exists, using well-defined staging and valuation rules. They should never mention the 3x, nor institutionalize any coverage ratio because, once you do so, you can be certain of only on thing: you will have that coverage ratio in your pipeline.

Enjoy your weekend!

I lived in west Cambridge when I was 4.

More recently, I lived in Brighton in my mid-twenties.

My daughter went to kindergarden in Brighton; my son was born at St Elizabeth's Hospital.

I worked on the east edge of the MIT campus and rode my bike to work along Western Avenue and through campus everyday.

My wife took the Green Line to the Back Bay to attend classes at Boston University.

I took the Red Line down to Dorchester to attend classes at UMass-Boston.

We used to go grocery shopping at the mall in Watertown.

Boston has had a Terrible, Horrible, No Good, Very Bad Week.

But Boston is a Very Wonderful Place, and I know it will recover.

Best wishes, and may the healing begin.

The last time I pulled off a Trifecta (3 ski resorts in 3 days), it was in 2010. I hadn't met Trish yet and DU Hockey had started their we're terrible in the playoffs streak. Last weekend, I decided to go for it again, this time with fiancé and kids in tow. We originally intended to ski Friday afternoon, but Abbie had an important "Mayor Election" at school and we were unable to leave early. By the time we arrived in Winter Park, the lifts were closed.

When we drove to Steamboat on Saturday, we only intended to ski 2 days. We dressed up in our outfits and got ready for the warm, mashed-potato conditions on Steamboat's closing weekend.

The conditions weren't great, so we only skied for a couple hours. Then we hung out with Trish's friends (Jenn and Todd) for the rest of the afternoon, swimming and frolicking with our kids like good parents do. On Sunday, we woke up and checked the ski report. While Steamboat did get 8" of fresh powder, Copper had 13 inches! So we packed up the car, puppies, and kids and drove 2 hours to Copper. When we arrived, we weren't disappointed. The kids were loving it and the conditions were great.

When we stopped for lunch, I was a bit disappointed that our ski weekend was about to end. Then I checked the Winter Park forecast, was pleasantly surprised, and asked "Hey, kids - should we complete the Trifecta and skip school tomorrow?" Of course, they said "YES" and our plans were set. We did a couple more runs, listened to some good live music at the base, then headed to our mountain views in Winter Park.

Sunday night is when the fun started. I didn't call the kids in sick for school, nor did I call in sick for my current client. I simply told them both we had a unique opportunity to complete a Trifecta and the snow was too good to leave.

Monday morning we woke up to 7" of fresh powder and by the time we got to Mary Jane, it was knee-deep. The runs were unbelievable. I skied several bump runs on Mary Jane without ever hitting bottom. It snowed hard all day long and by 2pm, they'd closed Berthoud Pass. This meant we had to stay another night. Some friends were stranded, so we offered them to stay with us and we enjoyed a nice evening reminiscing about all the fluffy snow and face shots. The first photo below is the snow accumulation on Monday morning; the second is from Monday at Happy Hour.

Skiing a Trifecta with my family seems like a perfect way to end the ski season. However, it's been snowing all week and I've had several powder runs since Monday. It's snowed 37 inches in the past 7 days and there's more snow in the forecast this weekend. Winter Park's Springtopia starts tomorrow. Live music, fresh powder, the Nuggets in the playoffs and a place at the base with good friends.

The fun ain't over yet!

This time it's personal. (I have lots of b-movie clichés, but I'll stop now...) The second candidate is now ready for voting.

Due to a few issues, I had to cancel the first 4.0.2 vote on Wednesday. Thanks to Sebastien and other folks who took the time to test the candidate and find issues!

That gave a little more time to fix a few more bugs, though it looks like we're not going to be getting CLOUDSTACK-528 fixed for this release.

Two new bug-fix releases of note in Apache security products:

Apache Santuario 1.5.4 has been released. Amongst the issues fixed is a thread-safety problem when secure validation is enabled, and a possible NPE due to ThreadLocal storage when an application is deployed in certain containers.

Apache WSS4J 1.6.10 has also been released. The issues fixed are available here. A performance issue was fixed in the MemoryReplayCache, which is used to guard against replay attacks. An interop issue was fixed with older Axis 1.x stacks. UsernameTokens with no password elements have been explicitly disallowed by default (although this is configurable). Finally, "time-to-live" functionality has been added to disallow "stale" UsernameTokens (with older Created values).

Passing test suites attest as to the quality of the code being tested. Just as importantly, it gives the developer confidence to evolve the code, i.e. to refactor or add new features without being afraid that the changes break existing functionality. Without the safety net provided by the test suite, evolving software can be a perilous and nerve wrecking activity to which few developers survive

A staircase in a multi-tenant house near the Malecón©James Duncan Davidson

There’s a distinct texture to Havana. It’d be easy to say that the city was frozen in time after the revolution and the embargo went into effect, but that’s not exactly the right way to describe it. It wasn’t hermetically sealed in some sort of bubble and perfectly preserved. Instead, it’s been lived in by generations of inhabitants who kept things going as best as they could, adding a coat of paint here or there when possible.

A wall inside a courtyard near the Malecón©James Duncan Davidson Two doorways©James Duncan Davidson

Given the relative disrepair of almost everything, there’s one thing about Cuba that I still find suprising: It’s not a place of squalor. Trash and litter collects in the buildings that have fallen apart too far and are abandonded, but anywhere people live is as tidy as one could expect. Certainly far tidier than anywhere else where people live on so little.

Pay phone on the streets of Havana©James Duncan Davidson

I have been waiting patiently for James Strachan to write a blog to introduce us to his latest project - hawtio. This project surely has the potential to become as great or even greater than Apache Camel.

In a one liner, hawtio is:

hawtio is a lightweight and modular HTML5 web console for managing your Java stuff.
Today James is busy doing the release of the hawtio 1.1. This is a very hawt release, that has great functionality already, so we should start spread the word about this project. We have also talked about doing small videos, as recoding what you do in a web browser is easy.

Anyway this blog entry today of mine, is not to steal the thunder from James. I would let him do the introduction of hawtio, and his vision for creating this project.

So in hawtio we have a Camel plugin, that allows you to gain insight into your running Camel applications. So the screenshot below is from hawtio running as a plain web application deployed to JBoss AS 7.1.1. You can use any web container of choice, such as Tomcat, Jetty etc. Though I was tasked to give the JBoss AS a little test spin, and thus had the opportunity to take a screenshot. I have also deployed the Apache Camel example camel-example-servlet-tomcat in the container (its also just a WAR application).

Dashboard in hawtio, with a live Camel diagram and table, updating in real time. hawtio allows you to create custom dashboards, where you can add any part of a view to the dashboard, and position the views as you please. So I created a Camel dashboard, which has a live diagram of the route from the camel-example-servlet-tomcat. As well a table with the running routes and their state and some key metrics. All the data on the dashboard updates in real time, which is very cool.

And the dashboards can be saved to a git repository, and you can share them with your team. And as its just a URL you can easily access the dashboard, by just copying the url from the browser address bar.

Anyway hawtio has a lot more cool stuff already. And we are having fun hacking on it. Its of course a 100% open source project, ASL licensed, and hosted at github. Everybody is welcome to contribute and hack on it.

Stay tuned, more blogs and videos should be coming in the future.

• Reality, Reactivity, Relevance and Repeatability in Java Application Profiling

  this product from JInspired appears to support runtime profiling of java apps with < 5% performance impact

  (tags: profiling performance java coding measurement)

• You Lookin’ At Me? Reflections on Google Glass

  ex-Nokia product design guru Jan Chipchase on Google Glass

  (tags: google privacy technology google-glass pervasive-computing life future)

• Not the ‘best in the world’ – The Medical Independent

  Debunking this prolife talking point:

  ‘Our maternity services are amongst the best in the world’. This phrase has been much hackneyed since the heartbreaking death of Savita Halappanavar was revealed in mid October. James Reilly and other senior politicians are particularly guilty of citing this inaccurate position. So what is the state of Irish maternity services and how do our figures compare with other comparable countries? Let’s start with the statistics. The bottom line: Eight deaths per 100,000 is not bad, but it ranks our maternity services far from the best in world and below countries such as Slovakia and Poland.

  (tags: pro-choice ireland savita medicine health maternity morbidity statistics)

• How Kaggle Is Changing How We Work – Thomas Goetz – The Atlantic

  Founded in 2010, Kaggle is an online platform for data-mining and predictive-modeling competitions. A company arranges with Kaggle to post a dump of data with a proposed problem, and the site’s community of computer scientists and mathematicians — known these days as data scientists — take on the task, posting proposed solutions. [...] On one level, of course, Kaggle is just another spin on crowdsourcing, tapping the global brain to solve a big problem. That stuff has been around for a decade or more, at least back to Wikipedia (or farther back, Linux, etc). And companies like TaskRabbit and oDesk have thrown jobs to the crowd for several years. But I think Kaggle, and other online labor markets, represent more than that, and I’ll offer two arguments. First, Kaggle doesn’t incorporate work from all levels of proficiency, professionals to amateurs. Participants are experts, and they aren’t working for benevolent reasons alone: they want to win, and they want to get better to improve their chances of winning next time. Second, Kaggle doesn’t just create the incidental work product, it creates a new marketplace for work, a deeper disruption in a professional field. Unlike traditional temp labor, these aren’t bottom of the totem pole jobs. Kagglers are on top. And that disruption is what will kill Joy’s Law. Because here’s the thing: the Kaggle ranking has become an essential metric in the world of data science. Employers like American Express and the New York Times have begun listing a Kaggle rank as an essential qualification in their help wanted ads for data scientists. It’s not just a merit badge for the coders; it’s a more significant, more valuable, indicator of capability than our traditional benchmarks for proficiency or expertise. In other words, your Ivy League diploma and IBM resume don’t matter so much as my Kaggle score. It’s flipping the resume, where your work is measurable and metricized and your value in the marketplace is more valuable than the place you work.

  (tags: academia datamining economics data kaggle data-science ranking work competition crowdsourcing contracting)

• The useful JVM options

  a good reference, with lots of sample output. Not clear if it takes 1.6/1.7 differences into account, though

  (tags: jvm reference java ops hotspot command-line)

The new Firefox OS phones from geeksphone are nearly here, with the announcement that next week they will be available from the currently-dormant online shop.

~

It was back in 2009 that I got a Palm Pre, perhaps the first commercial demonstration of “HTML for (almost) everything”. The turbulent history of that device is well-known, but shouldn’t detract from the core idea of building the entire user experience and applications using web technologies.

So I’m excited about Firefox OS, but thus far I’ve been sceptical about the platform’s ability to get developer attention, to build an ecosystem, and to reach a scale that makes the platform viable in the long-term. It depends very much on Mozilla’s and Telefonica’s ability to execute, and to get other device manufacturers to actually ship hardware.

~

As a small nitpicking example of execution, there’s no nice semantic URL I can point you to for Firefox OS. The best I could find is http://www.mozilla.org/en-US/firefox/partners/ or possibly http://www.mozilla.org/en-US/firefox/partners/#os. For developers it’s a little better, with https://developer.mozilla.org/en/docs/Mozilla/Firefox_OS.

I could talk about my disappointment with Mozilla‘s engagement with LiMo Foundation, where they had the opportunity to become the de facto web runtime as well as to demonstrate to mobile operators how to effectively work in an open source and open development model. If that had worked, we could have seen a precursor of today’s Firefox OS in 2009, rather than waiting four more years. Maybe some version of Firefox OS would be ascendant, and not the Android hydra. 

It’s churlish to criticise Mozilla alone, when the rest of the LiMo membership frittered away any opportunity for leadership in web runtimes because they couldn’t play nicely together, but I do think Mozilla missed a trick by not adopting a platform approach much sooner. Even as late as 2010 key people at Mozilla seemed to have their head in the clouds, when I had a heated discussion about the importance of web runtimes at GUADEC. I suspect that today’s Firefox OS is more the result of luck and external enthusiasm around B2G rather than any clearly-planned advance strategy.

Thankfully I think the team at Telefonica have a clear vision of what’s needed from Firefox OS, and with their experience of running “the thinking man’s developer engagement” (BlueVia), I suspect they also have a good idea of how to take it to the developer market.

~

One factor that’s changed my mind significantly about the likely success of Firefox OS was my trial of the Nokia Lumia. With a limited selection of apps in the Windows Phone app store, I was forced to rely rather more heavily on web apps for some core functionality and to fill the gaps, to get me through the day.

When the Palm Pre first launched, many websites were simply unusable from a mobile device. My time with the Lumia showed me how far along the mobile web has come (and how far it still lags behind), but one thing was clear: in the last few years, it’s now become possible to survive with web apps and a good mobile browser. Many companies need to address the desktop market with good websites even whilst building native apps. For those companies in particular, being Firefox OS-friendly is a simple step and a no-brainer.

~

What’s eye-catching about the geeksphone announcement is the pricing of the new devices: the Keon is €91, and the Peak is €149. That’s a great price for a smartphone. I was curious to see how the specifications stack up against similar phones, so I did a quick comparison using information from the encyclopaedic GSM Arena.

I compared published specifications against the last attempt at a “web runtime” mobile device, the HP Pre 3. I also compared against the iPhone 5 and iPhone 4S, the Samsung Galaxy SII / SIII / S4, and the Lumia 920.

The raw data is below in a spreadsheet, but from first glance these geeksphone handsets are not bad devices, with some obvious economic trade-offs – mostly around CPU and memory. When comparing prices you’ll probably want to factor in a good fast microSD card since the phones are light on storage. It will be interesting to see how well Firefox OS performs on those CPUs – but given announced future devices are even lower-spec, I would hope that lots of optimisation will make them sufficiently usable.

The Keon should be fine for development, but for normal use I’d most likely go for the Peak. Based on the specification, it seems like a realistic day-to-day device. At this price point, they are cheap enough to risk some money on for experimentation.

Of course the raw data doesn’t cover things like differences between chipsets in real life – see Why a Snapdragon S4 Galaxy SIII Is Awesome for an interesting discussion of this.

~

Finally – there’s a Firefox OS Hack Day coming up in the UK at the end of May – which should be a good opportunity to do some development with the devices.

~

The raw stats:

Note pricing data is approximate. If you notice any errors or can provide any more details, please let me know.

A couple on the streets watching people shop at an ad hoc market©James Duncan Davidson

During our pre-trip briefing the night before leaving for Cuba, David Hobby talked a bit about how he was going to approach the week photographically and what he hoped to see in the quick edit of six photos that we all would present at the end of our time in Havana. During his talk, he called me out saying, “Duncan goes to these incredible places and then posts all these beautiful photos that could be post-apocalyptic because they don’t have any people in them.” Then, he issued a challenge directly to me: “I want to see people in your shots this trip.”

Fair enough. He had a point. Loving a good challenge, my response was: “You’re on!”

Boys jumping into the water along the Malecón©James Duncan Davidson Along a street somewhere in Havana©James Duncan Davidson

It wasn’t all that hard, of course, to photograph people in Cuba. Cubans are friendly, and curious. Even the street hustlers wanting to take you to a bar for mojitos or a shop to get a cigar are happy to hang out, talk, and give you their perspective on life if you don’t say yes to their proposition. Sometimes that perspective was polished. Other times, not so much. We heard a lot of raw emotion about things that happened in the past.

Despite the ease with which we could mix with locals, it would have been all to easy to let the prescribed activities we participated in—we travelled on a person-to-person cultural exchange permit from the US Department of State and the Cuban tourist authorities have a vested interest in showing the most polished side of their country—to drive the photographic agenda. If that’d happened, we would all have come home with portfolios of old folks smoking cigars and hanging out in lovingly cared for decaying old houses. We wanted more than that.

So, we took full advantage of our free time outside the prescribed activities. We got up early, braved the hot and humid afternoons when the sun was at its strongest, and stayed up into the wee hours of the morning wandering streets—sometimes in parts of town that we were warned about. “Watch your camera!” was something that Bryan and I heard more than a few times as we wandered through the sketchier areas.

Sitting in a doorway in downtown Havana©James Duncan Davidson A couple walks along the street hand in hand©James Duncan Davidson

There are many that want to force one kind of narrative or another onto Cuba because of its past, its odd relationship with the USA as a result, and the interesting position it finds itself in right now as it transitions from whatever it was into whatever it becomes. Some of those narratives are fascinating. Others verge on shrill or even extremist. All of them have a place and deserve to be part of the conversation. If you stick too much to the traditional narratives, however, and focus solely on what was or how it came to be, you’ll miss the most important story right now: The transition in Cuba is in full swing.

I can’t begin to communicate how much has changed since I was last there two years ago on a marine science mission. Commerce has really ramped up. A lot of people are taking those small steps from selling a few goods to opening up shops to planning for a future. Every time I told someone that I was there before, they asked if I noticed the changes. How could one not? Even the food was better in quality on average everywhere we went.

Along a street somewhere in Havana©James Duncan Davidson A boxer prepares for practice©James Duncan Davidson

Have the Cuban people been through a lot? Very much so. Is it all suddenly great and rosy? Not even close. Things happen or don’t happen on a whim of persons unknown. Major problems, like pervasive street prostitution that is openly ignored by authorities, are easily visible. Will there continue to be problems going forward? Almost certainly. This won’t—and probably can’t—be a perfect process and there will be some big bumps as the gap between haves and have nots inevitably increases. But, almost everyone I met on the trip—certainly everyone I photographed—was participating in that process and looking forward to the future.

Of course, my viewpoint during my time in Cuba isn’t without bias. I was there as an outsider. If we chose, we could have stayed in a relatively insulated bubble drinking mojitos and smoking cigars that cost a month or more of a doctor’s salary—something like $25—while sitting under palm trees. Yes, we had mojitos and smoked a cigar or two, but we also got out and walked five to ten miles a day and saw as much as we could. The people in these photos are representative of who I saw on those walks.

Kids playing basketball and baseball in an abandoned lot©James Duncan Davidson

In order to keep the UK National Grid working correctly a delicate balancing act must be performed. The various units generating energy must be matched with the likely demand in order to keep the system within strict limits of frequency and voltage. When the system drifts beyond those limits bad things happen, inevitably depriving people of the energy they need.

Underpinning this process is a complex set of rules and procedures, detailed on the National grid website here. Given the scale of the UK energy market and the significance of the service, the volumes (and therefore cashflow) involved are very large. An obvious question when developing Variable Pitch was how involved wind energy producers were?

The information is available via the BMRS webiste (Elexon) which allows you to get figures for all Balancing Mechanism Units for any given date or settlement period. This had been available on Variable Pitch since Nov 2012 and showed the volumes and financial amounts involved. However, as with any set of numbers, taken out of context it was difficult to fully assess them. While the amounts paid to wind generators seem large – are they really out of proportion with those being paid to the other participants? Context is important and was lacking.

Once the lack of context had been pointed out I started looking at the available data again and started collecting additional information to allow me to provide the context that was lacking. Along the way I had to delve deeper into what the data meant in order to avoid misrepresenting it, but that required me to contact Elexon UK. Initially they were happy to answer questions, but they then delivered this statement.

The information is available, but the data is owned by Elexon and permission to reproduce will need to be obtained.

Variable Pitch has always had a strict policy about the data it uses.

Only data that is freely available and can therefore be validated from the source direct should be used.

It may not have been stated on the website (but perhaps it should be?), but that’s been the underlying policy I have always applied. All sources of data are noted on the site and no attempt is made to disguise where information comes from. As far as possible the “raw data” is available on the site. This is the minimum I feel is required for an open and honest representation.

I could contact Elexon UK and request permission to use the data, but then anyone else who wished to do likewise would need to contact them, which isn’t acceptable to me. Sadly I have reached the decision that I will have to remove all data that is sourced from their website.

To say I do this with a heavy heart is an understatement and it marks a retrograde step by decreasing the transparency and honesty with which the energy market operates. Elexon UK make money by selling the data so I can fully understand their desire to control the data, but as it relates to an industry that is an essential service it really should be freely available.

Without free and open access to such data honest, open and informed debate is impossible.

The Apache Airavata PMC is pleased to announce the immediate availability of the Airavata 0.7 release.

The release can be obtained from the Apache Airavata download page - http://airavata.apache.org/about/downloads.html
Release notes are available at - https://svn.apache.org/repos/asf/airavata/tags/airavata-0.7/RELEASE_NOTES
Apache Airavata is a software framework providing API’s, sophisticated server-side tools, and graphical user interfaces to construct, execute, control and manage long running applications and workflows on distributed computing resources. Apache Airavata builds on general concepts of service oriented computing, distributed messaging, and workflow composition and orchestration.
For general information on Apache Airavata, please visit the project website: http://airavata.apache.org/

• Austerity policies founded on Excel typo

  You’ve probably heard that countries with a high debt:GDP ratio suffer from slow economic growth. The specific number 90 percent has been invoked frequently. That’s all thanks to a study conducted by Carmen Reinhardt and Kenneth Rogoff for their book This Time It’s Different. But the results have been difficult for other researchers to replicate. Now three scholars at the University of Massachusetts have done so in “Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff” and they find that the Reinhart/Rogoff result is based on opportunistic exclusion of Commonwealth data in the late-1940s, a debatable premise about how to weight the data, and most of all a sloppy Excel coding error. Read Mike Konczal for the whole rundown, but I’ll just focus on the spreadsheet part. At one point they set cell L51 equal to AVERAGE(L30:L44) when the correct procuedure was AVERAGE(L30:L49). By typing wrong, they accidentally left Denmark, Canada, Belgium, Austria, and Australia out of the average. When you run the math correctly “the average real GDP growth rate for countries carrying a public debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0.1 percent.”

  (tags: austerity politics excel coding errors bugs spreadsheets economics economy)

• Is Your MySQL Buffer Pool Warm? Make It Sweat!

  How GroupOn are warming up a failover warm MySQL spare, using Percona stuff and a “tee” of the live in-flight queries. (via Dave Doran)

  (tags: via:dave-doran mysql databases warm-spares spares failover groupon percona replication)

It's taken a lot longer than I had planned, but the VOTE is now open for Apache CloudStack 4.0.2.

The CHANGES for 4.0.2 are up to date, I believe. As a maintenance release, there are no new features and should be no database changes or API changes – just pure bugfix goodness.

Please take a look at the release and give it a spin, the vote is open for a minimum of 72 hours. More eyes are always better. I'm hoping it'll be ready to roll on the first try, but I'd rather hear about any show stoppers or problems now rather than after we've done a release!

A 22 degree solar halo and the José Martí Memorial©James Duncan Davidson

While visiting the José Martí Memorial in Havana, Bryan Jones and I spotted a 22º solar halo in the sky around the sun caused by ice crystals high up in the atmosphere. They’re not exactly uncommon, but seeing a nice strong one like this was fun photographically.

A 22 degree solar halo as seen from the National Hotel in Havana©James Duncan Davidson

A bit later, while we were sipping mojitos at the Hotel Nacional de Cuba, we spotted the halo again—this time even more intensely and sporting an oval outside segment. We discussed the various permutations that would cause what we were seeing, but without easy access to the internet, we simply shrugged and made more photographs, as one does when in Cuba with a cold beverage and camera in hand.

• So now you know who gets some of those excessive Ticketmaster fees….

  Interesting evidence; it appears Irish music promoters are getting “rebates” from the massive TicketMaster “booking fee”, on each ticket sold. This sounds like a cartel to me, and we need to regulate this. Where is the National Consumer Agency and Competition Authority?

  The matter is something which should be of concern to every gig-going music fan, regardless of whether they go to Stradbally or not. For years, many have asked about TicketMaster’s quasi-monopoly position in the marketplace and why this is so. We’ve always been told that promoters preferred to deal with one company rather than several and that TM’s systems and nationwide reach yadda yadda yadda was the bees’ knees etc. Other companies have tried to compete but no-one has been able to beat TM at this game. But why would promoters go elsewhere when they’re getting a slice of the TM fees back as rebates? Those past off-the-record attempts by and briefings from promoters blaming TM for those fees can now be seen as hypocritical. They’re sticking with TM because they’re receiving a take of the fees paid by punters who have no other choice in service provider if they want to get their hands on tickets. You wonder what the acts make of this cash-grab – perhaps some whip-smart agent is already making a claim for a percentage of the rebates because there would be no rebates in the first place without the act. Surely this is an issue for the Competition Authority and National Consumers Association too, given the manner in which the rebates are made and TM’s deals with the promoters? While promoters under TM deals are free to sell a certain proportion of their tickets with another provider, it’s usually only a very small percentage of the total and unlikely to trouble TM’s bottom line. Also, given that the rebates are volume-driven, it’s better for the promoters to keep the largest possible chunk of their business with TM. It seems that we have a new suspect in the blame game about why ticket prices are so high.

  (tags: regulation ireland cartels competition ticketing tickets ticketmaster music gigs consumer)

• Blog shines spotlight on Dublin city’s illegal dumping problem

  Hooray, Eoin’s activism gets some coverage!

  THE SCALE OF Dublin’s dumping problem is laid bare in a blog that has seen contributors send in photos of chairs, fridges and heaps of rubbish strewn on city streets. Eoin Parker, one of organisers behind DublinLitterBlog.com, spoke to TheJournal.ie about the problem, saying that the blog was set up following the privatisation of waste management by Dublin City Council in 2012.

  (tags: dumping dublin litter rubbish blogs dcc d1 activism community)

• Ked

  To our knowledge, Ked is the first scripting language to emerge from The People’s Republic of Cork. Below is an account of what we know so far about the mysterious Corkonian language. Any suggested updates or contributions are encouraged. Genius.

  (tags: coding cork jokes funny like languages programming)

• Just how bad are RTE’s finances?

  A sobering examination by NAMAwinelake into the quagmire of Ireland’s publicly-funded national broadcaster:

  It seems that RTE has become a disaster zone, with libels and incompetence overseen by incapable management, and this is reflected in that organisation’s financial results. RTE still employs nearly 2,000 people and supports jobs and industry across independent producers and suppliers; it is a major business. But the time has come to call a halt to delusional management that is sinking the organization deeper into a quagmire which will ultimately need to be bailed out by the State. And Noel Curran is fobbing us off with flying a kite about a reduction in 65-year old Pat Kenny’s salary from €630,000 to €570,000?!

  (tags: rte namawinelake public funding finances money mismanagement ireland incompetence tv news)

• High Scalability – Scaling Pinterest – From 0 to 10s of Billions of Page Views a Month in Two Years

  wow, Pinterest have a pretty hardcore architecture. Sharding to the max. This is scary stuff for me:

  a [Cassandra-style] Cluster Management Algorithm is a SPOF. If there’s a bug it impacts every node. This took them down 4 times. yeah, so, eek ;)

  (tags: clustering sharding architecture aws scalability scaling pinterest via:matt-sergeant redis mysql memcached)

I wish I could write like Shakespeare but since I don't you are left with this blog about Representational State Transfer (REST) and specifically a discussion on whether the CloudStack API is a REST API or not. The short answer is that the CloudStack API is RESTlike but not RESTfull since it is only based on the GET method. Being an http based API that can return JSON does not make it a RESTfull API. This should not be seen as negative criticism but just a clarification.

A few words first about the CloudStack API and a few pointers. It is very extensive, from user creation, vm management to firewall configuration and more advanced networking features. Github has lots of clients written by the community and I am sure you can find your favorite language in there. The Developer Guide explains in details how to make requests. With CloudMonkey interacting with the API has become even easier. It is a terrific way to learn the API and check the required parameters of each call. Coupled with devcloud you can have a fully functional local CloudStack testbed.

To explore the API, I often also look at the GUI and see how the API calls are made. To do this, I access the UI in Firefox and launch Firebug console. I can then see the calls that I am interested in and specifically check the parameters/headers etc or the http call being made.

REST is known as an architectural style and to put it in french-english it is basically a way to design your API to create a web service that only uses the HTTP methods to manipulate the state of web resources. In my opinion it was really a response to the complexity of the SOAP based web services and the many standards that came out. REST really took off with Web 2.0 and was seen as a way to create easy to use web services. I recently found a few articles by Luis Rei and they clearly explain how to design a REST web service as well as implement on with Flask a lightweight python based web framework.

Looking closely at the CloudStack API we see that it uses http, can return JSON objects but only uses the GET method of HTTP. As such it is really a Query API that we can call RESTLike but not RESTfull similarly to the Amazon AWS EC2 Query API.

To illustrate this point I decided to write a REST wrapper on top of the CloudStack API. This was really the reason behind my CloudStack Silly Tuesday Hack. The actual usefulness of it is still questionable :) it would be better to re-write the CloudStack API into a REST native API. That being said "Cloud wrappers" like jclouds/deltacloud have adopted a stance where they expose an API on top of existing IaaS APIs. So who knows it may be useful.

First I wanted to be able to make http calls via curl and use the default GET based CloudStack API, then I wanted to make it more RESTFull. I decided to use Flask because it has a clean way to define web routes and specify the HTTP method being used. To do all of this I used DevCloud as mentioned above and I was checking all my calls with CloudMonkey. To make simple GET calls via curl and avoid constructing the signature of the calls, I took the requester.py file from CloudMonkey and imported it in my Flask application, checkout the gist for the entire code. Below is a sample "route". In this example we only consider user management.

@app.route('/list')
def list():
print request.query_string
res={}
for key in request.args.iterkeys():
res[key]=request.args.get(key)
print res
response, error = requester.make_request('listUsers',res,None,host,port,apikey,secretkey,protocol,path)
return response

An http GET to http://localhost:5000/list will be routed to the list() function. The query parameters will be parsed and stored in a dictionary which in turn will be given to listUsers from CloudStack. This is where the silliness lies. A GET that does a GET. Using curl we can easily call this method:

curl -X GET -G 'http://localhost:5000/list'
{ "listusersresponse" : { "count":4 ,"user" : [ {"id":"7ed6d5da-93b2-4545-a502-23d20b48ef2a","username":"admin","firstname":"admin","lastname":"cloud","created":"2012-07-05T12:18:27-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg","secretkey":"VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"1fea6418-5576-4989-a21e-4790787bbee3","username":"runseb","firstname":"sebgoa","lastname":"goa","email":"joe@smith.com","created":"2013-04-10T16:52:06-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"Xhsb3MewjJQaXXMszRcLvQI9_NPy_UcbDj1QXikkVbDC9MDSPwWdtZ1bUY1H7JBEYTtDDLY3yuchCeW778GkBA","secretkey":"gIsgmi8C5YwxMHjX5o51pSe0kqs6JnKriw0jJBLceY5bgnfzKjL4aM6ctJX-i1ddQIHJLbLJDK9MRzsKk6xZ_w","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"b3b60a8d-df6f-4ce6-a6f9-6194907457a5","username":"john","firstname":"sebgoa","lastname":"goa","email":"runseb@gmail.com","created":"2013-04-12T05:09:10-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"62d866e4-da97-46c2-a3e1-10faf6197c73","username":"titi","firstname":"www","lastname":"rr","email":"joe@smith.com","created":"2013-04-15T06:21:59-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

In the response we see the list of all the users in the system. The same list route would also work if we were to pass the id of a single user:

curl -X GET -G 'http://localhost:5000/list' -d id=7ed6d5da-93b2-4545-a502-23d20b48ef2a
{ "listusersresponse" : { "count":1 ,"user" : [ {"id":"7ed6d5da-93b2-4545-a502-23d20b48ef2a","username":"admin","firstname":"admin","lastname":"cloud","created":"2012-07-05T12:18:27-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg","secretkey":"VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

Where it becomes interesting is that deleting a user will be the same code but a different API call:

@app.route('/delete')
def delete():
print request.query_string
res={}
for key in request.args.iterkeys():
res[key]=request.args.get(key)
print res
response, error = requester.make_request('deleteUser',res,None,host,port,apikey,secretkey,protocol,path)
return response

You would call it to delete a user by passing the id of the user like this:

curl -X GET -G 'http://localhost:5000/delete' -d id=62d866e4-da97-46c2-a3e1-10faf6197c73
{ "deleteuserresponse" : { "success" : "true"} }

You are deleting a resource and yet, it is still a GET call. Why not using a DELETE call to delete a resource ? This would be more intuitive and a nice use of the HTTP grammar. The same situation will occur for updating a user or creating a user, still a GET while we could use a POST and a PATCH. In CloudStack-Flask I added some routes to create a proper REST service. For example a route becomes:

@app.route('/user/', methods=['GET','DELETE','PATCH'])
def user(uuid):
if request.method =='GET':
response, error = requester.make_request('listUsers',{'id':uuid},None,host,port,apikey,secretkey,protocol,path)
return response
elif request.method =='PATCH':
data = request.json
data['id']=uuid
response, error = requester.make_request('updateUser',data,None,host,port,apikey,secretkey,protocol,path)
return response
else:
response, error = requester.make_request('deleteUser',{'id':uuid},None,host,port,apikey,secretkey,protocol,path)
return response

The id of the user is specified in the URL. The user is the actual web resource whose state we are changing. And the HTTP method is used to determine the type of action. In our case, the GET lists the user information, the PATCH updates the parameters of the user and the DELETE deletes the user. The id is now part of the URI and not a query parameters

curl -X GET http://localhost:5000/user/b3b60a8d-df6f-4ce6-a6f9-6194907457a5
{ "listusersresponse" : { "count":1 ,"user" : [ {"id":"b3b60a8d-df6f-4ce6-a6f9-6194907457a5","username":"john","firstname":"sebgoa","lastname":"goa","email":"runseb@gmail.com","created":"2013-04-12T05:09:10-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

curl -X DELETE http://localhost:5000/user/b3b60a8d-df6f-4ce6-a6f9-6194907457a5
{ "deleteuserresponse" : { "success" : "true"} }

curl -X PATCH -H "Content-Type: application/json" http://localhost:5000/user/1fea6418-5576-4989-a21e-4790787bbee3 -d '{"firstname":"foobar"}'
{ "updateuserresponse" : { "user" : {"id":"1fea6418-5576-4989-a21e-4790787bbee3","username":"runseb","firstname":"foobar","lastname":"goa","email":"joe@smith.com","created":"2013-04-10T16:52:06-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"Xhsb3MewjJQaXXMszRcLvQI9_NPy_UcbDj1QXikkVbDC9MDSPwWdtZ1bUY1H7JBEYTtDDLY3yuchCeW778GkBA","secretkey":"gIsgmi8C5YwxMHjX5o51pSe0kqs6JnKriw0jJBLceY5bgnfzKjL4aM6ctJX-i1ddQIHJLbLJDK9MRzsKk6xZ_w","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} } }

Identify your resources/entities and use the HTTP grammer to modify their state. Don't forget to check my cloudstack-flask app, it's very basic right now but could be a nice REST wrapper doubled-up with a bootstrap based UI, we will talk about the web views in the next post.