Planet Apache

Syndicate content
Updated: 1 day 7 hours ago A guide to Apache's Spark Streaming

Thu, 2015-04-23 02:00

Apache Spark is an open source cluster computing framework. In contrast to Hadoop’s two-stage disk-based MapReduce paradigm, Spark’s in-memory primitives provide performance up to 100 times faster for certain applications.

Categories: FLOSS Project Planets

Bryan Pendleton: Fess up

Wed, 2015-04-22 21:19


Come on, admit it.

Be honest.

We just don't have a freaking clue.

  • Delayed flash crash arrest may herald future spoofing detection woesTim Massad, the head of the Commodity Futures Trading Commission, which oversees the trading of futures and swaps, said on Wednesday that it took so long to charge Sarao because of the size and complexity of U.S. derivatives markets. "These are huge markets," he said. "There's a lot going on."
  • Trading at the speed of lightThe faster your trading system, the quicker you can take advantage of those discrepancies. But that is not all the high-frequency traders can do. They can “front-run” news events, jumping on anything that might move a price, and get in and out of a stock before anyone else knows that something has happened. By trading at such speed, they can even get into an equity in the gap between the placing of a large buy order and the execution of it, and thereby buy just before the price rises.
  • ‘Flash crash’ charges spark alarm over regulation of US marketsSherrod Brown, the top Democrat on the Senate banking committee, told the Financial Times: “It’s encouraging that the Justice Department and [Commodity Futures Trading Commission] are pursuing this case, but troubling that it has only come to light now with the help of a whistleblower who invested substantial time in putting the pieces together.”
  • 'Flash Crash' arrest shakes investors' confidenceAdding to concerns, the CFTC was alerted to Sarao's alleged misdeeds by a whistle-blower, who has not been identified, according to Shayne Stevenson, who represents the whistle-blower through Hagens Berman law firm in Seattle. Stevenson said his client brought "high-quality information" about "market manipulation" to the CFTC, which alerted the DOJ.
  • Accused British 'flash crash' trader fights extradition to U.S.A British man accused of market manipulation that contributed to the May 2010 Wall Street "flash crash" said he opposed being extradited to the United States, while the operator of the market where he traded sought to rebut prosecutors' suggestion that futures helped cause the crash.
  • Roots of 'flash crash' go back further than you thought at CMEMore than a year before the May 6, 2010 "flash crash," CME Group noticed questionable trading in its E-mini market by a particular electronic trader who was placing orders and cancelling them.

    As the crash whipsawed the futures and stock markets in 2010, CME saw the suspicious activity again and warned the trader that day that orders must be placed “in good faith,” without an intent to cancel. The trader responded two weeks later: “Kiss my ass.”

  • How computerized trading in the hands of a nobody in Britain allegedly crashed the stock marketNot everyone agrees.
  • A Sweatpants-Wearing Rando Might Have Caused the Flash CrashHe seems to have executed his trades out of a modest, semi-detached house under the Heathrow flight path that he shared with his parents. He used off-the-shelf software that he souped up to make his bets. He named one shell company "Nav Sarao Milking Markets Ltd." The guy showed up to court in a pair of white sweatpants.
  • 'Flash Crash' Arrest Raises More Questions Than AnswersBut surely, since then regulators went through the action that took place on and around the day of the Flash Crash with a very fine comb? And if so, why did Sarao's alleged actions not jump at them as strange at the time?
  • Why Did It Take So Long To Catch The ‘Flash Crash’ Futures Trader?The CME Group declined to comment, citing the ongoing investigation. But analysts worry that the CME’s revenue model interferes with its motivation to police trading. The more trades that zip through the exchange, the more money it makes. That means it could be disincentivized from tackling manipulative traders who still bring valuable liquidity to the market.
  • Spoofing the Stock Market -- Here’s One Way to Stop ItIt may be a bit more complicated than that, but not much.
  • Junk bonds really to blame for Flash CrashJunk Debt collapsed a full hour before the stock market Flash Crash took place (and I would add that utilities and Treasurys were leading prior, indicating a VIX spike was possible). If junk debt collapses, how can stocks not collapse afterward when in a bankruptcy proceeding, junk debt has a higher claim on assets than equity? What caused the Flash Crash in stocks was a Flash Crash in junk debt, which was like a 1987 style crash in credit.

It's not like this is anything important.

It's not like it's the entire world economy.

Who cares whether we understand it or not?

Oh, dear.

Categories: FLOSS Project Planets

Justin Mason: Links for 2015-04-22

Wed, 2015-04-22 18:58
  • ‘Continuous Deployment: The Dirty Details’

    Good slide deck from Etsy’s Mike Brittain regarding their CD setup. Some interesting little-known details: Slide 41: database schema changes are not CD’d — they go out on “Schema change Thursdays”. Slide 44: only the webapp is CD’d — PHP, Apache, memcache components (, support and back-office tools, developer API, gearman async worker queues). The external “services” are not — databases, Solr/JVM search (rolling restarts), photo storage (filters, proxy cache, S3), payments (PCI-DSS, controlled access). They avoid schema changes and breaking changes using an approach they call “non-breaking expansions” — expose new version in a service interface; support multiple versions in the consumer. Example from slides 50-63, based around a database schema migration. Slide 66: “dev flags” (rollout oriented) are promoted to “feature flags” (long lived degradation control). Slide 71: some architectural philosophies: deploying is cheap; releasing is cheap; gathering data should be cheap too; treat first iterations as experiments. Slide 102: “Canary pools”. They have multiple pools of users for testing in production — the staff pool, users who have opted in to see prototypes/beta stuff, 0-100% gradual phased rollout.

    (tags: cd deploy etsy slides migrations database schema ops ci version-control feature-flags)

  • Etsy’s Release Management process

    Good info on how Etsy use their Deployinator tool, end-to-end. Slide 11: git SHA is visible for each env, allowing easy verification of what code is deployed. Slide 14: Code is deployed to “princess” staging env while CI tests are running; no need to wait for unit/CI tests to complete. Slide 23: smoke tests of pre-prod “princess” (complete after 8 mins elapsed). Slide 31: dashboard link for deployed code is posted during deploy; post-release prod smoke tests are run by Jenkins. (short ones! they complete in 42 seconds)

    (tags: deployment etsy deploy deployinator princess staging ops testing devops smoke-tests production jenkins)

  • Makerbot’s Saddest Hour | TechCrunch

    I’ve been speaking to a few people [at Makerbot] who prefer to remain anonymous and most of my contacts there are gone (the head of PR was apparently fired) and don’t want to talk. But the new from inside is troubling. The mass-layoffs are blamed on low revenue and one former employee wrote “Company was failing. Couldn’t pay vendors, had to downsize.” Do I think Makerbot will sink? At this point I don’t know.

    (tags: makerbot 3d-printing startups downsizing layoffs ouch)

  • credstash

    ‘CredStash is a very simple, easy to use credential management and distribution system that uses AWS Key Management System (KMS) for key wrapping and master-key storage, and DynamoDB for credential storage and sharing.’

    (tags: aws credstash python security keys key-management secrets kms)

  • Lessons Learned while Working on Large-Scale Server Software”> -> Lessons Learned while Working on Large-Scale Server Software

    Good advice

    (tags: distributed scalability systems coding server-side erlang devops networking reliability)

Categories: FLOSS Project Planets

Colm O hEigeartaigh: Apache Santuario - XML Security for Java 2.0.4 released

Wed, 2015-04-22 08:09
Apache Santuario - XML Security for Java 2.0.4 has been released. The issues fixed are available here. Perhaps the most significant issue fixed is an interop issue which emerged when XML Security is used with OpenSAML (see the Apache CXF JIRA where this was raised).
Categories: FLOSS Project Planets

Justin Mason: Links for 2015-04-21

Tue, 2015-04-21 18:58
Categories: FLOSS Project Planets

Bryan Pendleton: Flash Crash news

Tue, 2015-04-21 13:44

Well, this is interesting: CFTC Charges U.K. Resident Navinder Singh Sarao and His Company Nav Sarao Futures Limited PLC with Price Manipulation and Spoofing.

In particular, the CFTC release notes:

in or about June 2009, Defendants modified a commonly used off-the-shelf trading platform to automatically simultaneously “layer” four to six exceptionally large sell orders into the visible E-mini S&P central limit order book (the Layering Algorithm), with each sell order one price level from the other. As the E-mini S&P futures price moved, the Layering Algorithm allegedly modified the price of the sell orders to ensure that they remained at least three or four price levels from the best asking price; thus, remaining visible to other traders, but staying safely away from the best asking price. Eventually, the vast majority of the Layering Algorithm orders were canceled without resulting in any transactions. According to the Complaint, between April 2010 and April 2015, Defendants utilized the Layering Algorithm on over 400 trading days.

The Complaint alleges that Defendants often cycled the Layering Algorithm on and off several times during a typical trading day to create large imbalances in the E-mini S&P visible order book to affect the prevailing E-mini S&P price. Defendants then allegedly traded in a manner designed to profit from this temporary artificial volatility. According to the Complaint, from April 2010 to present, Defendants have profited over $40 million, in total, from E-mini S&P trading.

As others quickly pointed out, the notion that "layering" is involved in these wild price swings is being studied by multiple agencies. For example: Exclusive: SEC targets 10 firms in high frequency trading probe - SEC document.

The SEC has been seeking evidence of abuse of order types, as well as traditional forms of abusive trading like "layering" or "spoofing" and other issues relating to high-frequency trading that might be violations of the law, SEC Director of Enforcement Andrew Ceresney told Reuters in May (

Spoofing and layering are tactics where traders places orders that they cancel before they are executed to create the false impression of demand, aiming to trick others into buying or selling a stock at the artificial price.

I'm pleased that investigators continue to investigate.

On the other hand, even after 5 years the investigators still appear to be uncertain as to exactly what happened and why.

It's disturbing news, all around.

Categories: FLOSS Project Planets

Justin Mason: Links for 2015-04-20

Mon, 2015-04-20 18:58
  • FBI admits flaws in hair analysis over decades

    Wow, this is staggering.

    The Justice Department and FBI have formally acknowledged that nearly every examiner in an elite FBI forensic unit gave flawed testimony in almost all trials in which they offered evidence against criminal defendants over more than a two-decade period before 2000. [....] The review confirmed that FBI experts systematically testified to the near-certainty of “matches” of crime-scene hairs to defendants, backing their claims by citing incomplete or misleading statistics drawn from their case work. In reality, there is no accepted research on how often hair from different people may appear the same. Since 2000, the lab has used visual hair comparison to rule out someone as a possible source of hair or in combination with more accurate DNA testing. Warnings about the problem have been mounting. In 2002, the FBI reported that its own DNA testing found that examiners reported false hair matches more than 11 percent of the time.

    (tags: fbi false-positives hair dna biometrics trials justice experts crime forensics inaccuracy csi)

  • The missing MtGox bitcoins

    Most or all of the missing bitcoins were stolen straight out of the MtGox hot wallet over time, beginning in late 2011. As a result, MtGox operated at fractional reserve for years (knowingly or not), and was practically depleted of bitcoins by 2013. A significant number of stolen bitcoins were deposited onto various exchanges, including MtGox itself, and probably sold for cash (which at the bitcoin prices of the day would have been substantially less than the hundreds of millions of dollars they were worth at the time of MtGox’s collapse). MtGox’ bitcoins continuously went missing over time, but at a decreasing pace. Again by the middle of 2013, the curve goes more or less flat, matching the hypothesis that by that time there may not have been any more bitcoins left to lose. The rate of loss otherwise seems unusually smooth and at the same time not strictly relative to any readily available factors such as remaining BTC holdings, transaction volumes or the BTC price. Worth pointing out is that, thanks to having matched up most of the deposit/withdrawal log earlier, we can at this point at least rule out the possibility of any large-scale fake deposits — the bitcoins going into MtGox were real, meaning the discrepancy was likely rather caused by bitcoins leaving MtGox without going through valid withdrawals.

    (tags: mtgox bitcoin security fail currency theft crime btc)

  • Bank of the Underworld – The Atlantic

    Prosecutors analyzed approximately 500 of Liberty Reserve’s biggest accounts, which constituted 44 percent of its business. The government contends that 32 of these accounts were connected to the sale of stolen credit cards and 117 were used by Ponzi-scheme operators. All of this activity flourished, prosecutors said, because Liberty Reserve made no real effort to monitor its users for criminal behavior. What’s more, records showed that one of the company’s top tech experts, Mark Marmilev, who was also arrested, appeared to have promoted Liberty Reserve in chat rooms devoted to Ponzi schemes. (via Nelson)

    (tags: scams fraud crime currency the-atlantic liberty-reserve ponzi-schemes costa-rica arthur-budovsky banking anonymity cryptocurrency money-laundering carding)

  • I was a Lampedusa refugee. Here’s my story of fleeing Libya – and surviving

    ‘The boy next to me fell to the floor and for a moment I didn’t know if he had fainted or was dead – then I saw that he was covering his eyes so he didn’t have to see the waves any more. A pregnant woman vomited and started screaming. Below deck, people were shouting that they couldn’t breathe, so the men in charge of the boat went down and started beating them. By the time we saw a rescue helicopter, two days after our boat had left Libya with 250 passengers on board, some people were already dead – flung into the sea by the waves, or suffocated downstairs in the dark.’

    (tags: lampedusa migration asylum europe fortress-europe italy politics immigration libya refugees)

  • Run your own high-end cloud gaming service on EC2

    Using Steam streaming and EC2 g2.2xlarge spot instances — ‘comes out to around $0.52/hr’. That’s pretty compelling IMO

    (tags: aws ec2 gaming games graphics spot-instances hacks windows steam)

  • Running Arbitrary Executables in AWS Lambda

    actually an officially-supported mode. huh

    (tags: lambda aws architecture ops node.js javascript unix linux)

Categories: FLOSS Project Planets

Jim Jagielski: Open Source Has Won The Battle; Let's Not Lose The War

Mon, 2015-04-20 10:01

The below is an abstract for a talk...

Open Source Has Won The Battle; Let's Not Lose The War

20 years ago, a bunch of us got together and created Apache, and then 5 years later went ahead and created The Apache Software Foundation. The idea of Open Source back then was weird, and wild, and suspect. But due to the power and capability of the Apache Web Server, in combination with Linux, Open Source gained traction, acceptance and now ubiquity.

Looking around at the IT landscape nowadays, Open Source is found everywhere. Software is eating the world, and Open Source is the utensil of choice. Corporations once critical of Open Source, now embrace it; Open Source is now both strategic and mandatory. In many ways, one could assume that Open Source has won.

Well, maybe it has won, but it's just won the battle; the war is still there, and our success in winning the battle is threatening to cause our loss of the war. 

"It's on Github, so of course it's Open Source, right?" Wrong.

"It's got an OSI license, so nothing else is needed, right?" Wrong.

"There's nothing wrong with paid developers/contributors, right?" Well... maybe yes and maybe no.

"What is really the matter with pay-to-play Open Source foundations?"  Give me 30 minutes or so, and I'll tell you what the risks are.

There's an old saying that in Open Source, developers/contributors scratch their own itches. But what about today? Do they still? Can they still? And what is the ultimate harm if they can't. And as more and more Open Source gets funded, directly, by corporations, where does that leave the true volunteer contributor? And finally, who really has the ultimate control over a project's destiny?

This presentation will give a short history of the Open Source movement, and why the most critical forces behind its success in being an innovation engine may be at risk. 

Categories: FLOSS Project Planets

Tom White: The Hay Dark Skies Festival, Reverend Thomas William Webb, and Jupiter

Sun, 2015-04-19 12:09
In 2013, the Brecon Beacons was designated a Dark Sky Reserve, and a year later the first Dark Skies Festival was held in Hay-on-Wye. The second festival took place this weekend, and my family went along to some of the activities.

Young stargazers, Lottie and MillieIn the morning, we found ourselves in a planetarium tent, then we looked at sunspots, and held pieces of meteorite.

The evening event was stargazing at Holy Trinity Church in Hardwicke, just outside Hay. Quite apart from the lack of light pollution, the location was a special one, since the vicar of the parish from 1856 until 1885 was Reverend Thomas William Webb, who in his spare time observed the night sky with telescopes and an observatory he had built himself.
Holy Trinity Church, Hardwicke
In 1859, while at Hardwicke he wrote the classic book, Celestial Objects for the Common Telescope, the object of which was "to furnish the possessors of ordinary telescopes with plain directions for their use, and a list of objects for their advantageous employment".

The book remained in print well into the following century (and was recently republished by Cambridge University Press), and it's probably difficult to overemphasise the importance of this book in encouraging generation after generation of amateur stargazers.

In the words of Janet and Mark Robinson, who used to live in the vicarage and have edited a book about Webb,
Like Patrick Moore, he was an enthusiast who wanted to inspire as many people as possible to look through a telescope. Even at the choir party he "arranged the telescope and acted as showman and all in turn had a look at Saturn".Webb would no doubt have been pleased to see yesterday's gathering of enthusiastic amateurs (including the Robinsons) with an impressive range of telescopes, on a cold but very clear night. The highlight for us was seeing Jupiter and its four brightest moons (Io, Europa, Ganymede and Callisto) through a large reflecting telescope. We could even see the north and south belts, and the Great Red Spot (or Pink Splodge as Lottie named it).

Sunset. Venus is visible top centreThank you to the organisers of the Hay Dark Skies Festival, and the volunteers from the Usk Astronomical Society (the oldest astronomical society in the UK), the Abergavenny Astronomy Society and the Heads of the Valleys Astronomical Society.
Categories: FLOSS Project Planets

Justin Mason: Links for 2015-04-18

Sat, 2015-04-18 18:58
Categories: FLOSS Project Planets

Bryan Pendleton: A hodge of podge

Sat, 2015-04-18 13:25

Boy, my mind has really been wandering recently.

Must be the weather.

  • The Town That Creep Built“Fault creep” is a condition that results when the underlying geology is too soft to get stuck or to accumulate tectonic stress: in other words, the deep rocks beneath Hollister are slippery, more pliable, and behave a bit like talc. Wonderfully but unsurprisingly, the mechanism used to study creep is called a creepmeter.

    The ground sort of oozes past itself, in other words, a slow-motion landslide at a pace that would be all but imperceptible if it weren’t for the gridded streets and property lines being bent out of shape above it.

  • SevenevesThe moon blew up without warning and for no apparent reason. It was waxing, only one day short of full. The time was 05:03:12 UTC. Later it would be designated A+0.0.0, or simply Zero.
  • Linux 4.0But "Hurr durr I'ma sheep" trounced "I like online polls" by a 62-to-38% margin, in a poll that people weren't even supposed to participate in. Who can argue with solid numbers like that? 5,796 votes from people who can't even follow the most basic directions?
  • Linux 4.0 releasedLooking at just the statistics in git, this release is not just when we cross half a million commits total, but also cross the 4 million git object limit. Interestingly (if you look for numeric patterns), Linux 3.0 was when we crossed a quarter million commits and 2 million git objects, so there's a nice (and completely unintentional) pattern there when it comes to the kernel git repository.
  • Almonds in California: They use up a lot of water, but they deserve a place in California's future.The problem is that, thanks to the current drought, the water supply is going away faster than expected. The almond industry is an indicator of how difficult it might be to adapt to climate change, economically and environmentally.

    What we’re witnessing in California right now is a glimpse into the future. California has now endured drought in 11 of the last 15 years, and there’s every reason to believe this is just the beginning.

  • Math for eight-year-olds: graph theory for kids!This morning I had the pleasure to be a mathematical guest in my daughter’s third-grade class, full of inquisitive eight- and nine-year-old girls, and we had a wonderful interaction. Following up on my visit last year (math for seven-year-olds), I wanted to explore with them some elementary ideas in graph theory, which I view as mathematically rich, yet accessible to children.
  • Replicating SQLite using Raft ConsensusI decided to build a distributed replication layer using the Raft consensus protocol, which gives me effective replication without the hassle of running a much heavier solution like MySQL. It provides all the advantages of replication, with the data modelling functionality of a relational database, but with the convenience of a single-file database.
  • A Million Little BoxesWhile crossword construction is unavoidably linguistic, constructors these days rely heavily on data and programming. Take, for instance, Puzzle 5, always the hardest of the tournament. This year, Jeff Chen was the sadist behind the curtain. An aspiring novelist from Seattle, he runs the crossword database XWord Info and has authored or co-authored 37 New York Times puzzles.

    Chen’s Puzzle 5 was titled “Attention, Newbies!” The conceit was adding new B’s (get it?) to familiar phrases. “Vocalist” became “vocab list,” “alloy” became “ballboy,” and so on. To find workable and interesting phrases like this, Chen wrote a Python script and applied it to his master word and phrase list. Using this program, he realized “caroms” could become “car bombs,” for example.

  • From the vault: Watching (and re-watching) “The Mother of All Demos”To give an idea of the scope of the demo, Engelbart demonstrated an early look at word processing, windowing, hypertext, and dynamic file linking, as well as using graphics in a computer program. It was also the first time many of the attendees had seen a mouse, although work on the mouse began in 1963.
  • Arcology: cutaways of the future city-hives that never wereSoleri taught architecture at ASU, but his real classroom was Arcosanti: "an experimental town and molten bronze bell casting community" a few hours from Phoenix, which was supposed to have 5,000 residents by now, demonstrating "lean" city living in high-density complexes that combined super-efficient usage of space with stylized, naturalistic exteriors that made each building part of the landscape.
  • The Cult of Work You Never Meant to JoinI take walks. I leave my phone in my pocket when I’m out with friends or eating my meals. I spend a fair amount of time on my hobbies, like writing and hunting for the world’s best cheeseburger.
  • Code QualityIt's like a salad recipe written by a corporate lawyer using a phone auto-correct that only knew Excel formulas.
Categories: FLOSS Project Planets

Rich Bowen: Twine

Sat, 2015-04-18 09:38

This week we did ApacheCon in Austin. I shipped the original Apache feather to the venue for 20th birthday of the Apache web server project, and it hung above the stage for the keynotes.

It’s an item that we’re very proud of, and of some historical significance.

The conference producers treated it like it was the Declaration of Independence or something. They handled it carefully and reverently.

At the end of the event the guy in charge of A/V came to me with some twine.

He said he had removed it from the hanging hooks on the feather in order to use black nylon that matched the stage dressing, and which would hang more securely. But he saved these scraps of twine because he knew how significant the item was to us.

Now, it’s not that the twine mattered – it was something I added years after the original was made. It’s that he cared enough, and respected our heritage enough to save it and track me down, that impressed me so very much. It really put a wonderful final touch on an almost-perfect event.

And this is why, among many other reasons, we love our conference production company, The Linux Foundation.

Categories: FLOSS Project Planets

Steve Loughran: Build tools as Proof Engines

Sat, 2015-04-18 08:08

Someone has put up a thoughtful post on whether you can view make as a proof system. I thought about it and have come up with a conclusion: maybe in the past —but not any more.

As background, it's worth remembering
  1. I did "write the book on Ant", which is a build system in which you declare the transformational functions required to get your application into a state which you consider shippable/deployable, a set of functions which you define in a DAG for the Ant execution engine to generate and order from and then apply to your environment
  2. Early last year, in my spare time, I formally specified the Hadoop FS APIs in Spivey's Z notation, albeit in a Python syntax.
  3. In YARN-913 I went on to specify the desired behaviour of the YARN registry as a TLA+ specification.
And while I don't discuss it much, during my undergraduate work on Formal Specification of and implementation Microprocessors, I was using Gordon's HOL theorem prover. The latter based on Standard ML, just to fend off anyone who believed me when I was claiming not to understand functional programming earlier this week. I didn't meant it, it just entertained the audience. Oh, and did I mention I'm happy to write Prolog too?

This means that (a) I believe I understand about specifications, (b) I vaguely remember what a proof engine is, and (b) I understand how Prolog resolves things, and specifically, why "!" is the most useful operator when you are trying to write applications.

Now I note that the author of that first post, Bob Atkey, does not only has a background of formal logic, SML and possibly even worked with the same people as me, his knowledge is both greater and more up to date than mine. I just have more experience of breaking builds and getting emails from jenkins telling me this.

Now, consider a software project's build
  1. A set of source artifacts, S, containing artifacts
  2. A declaration of the build process, B
  3. A set of external dependencies, libraries, L.
  4. A build environment, E., comprising the tools needed for the build, and the computer environment around them, including the filesystem, OS, etc.
The goal is to reach a desired state of a set of redistributables, R, such that you are prepared to ship or deploy them.

The purpose of the build system, then, is to generate R from S through a series of functions applied to (S, L) with tools T within the environment E. The build process description, B, defines or declares that process.

There's many ways to do that; a single line bash file cc *.c && cc *.o could be enough to compile the lexical analyser example from Ch02 of the dragon book.

Builds are more complex than that, which is where tools like make come in.

Make essentially declares that final list of redistributables, R, and a set of transformations from inputs artifacts to output artifacts, including the functions (actions by the tools) to generate the output artifacts from the input artifacts.

The make runtime looks at what artifacts exist, what are missing, and what are out of date somehow builds a chain of operations that are hypothesised to produce a complete set of output artifact whose timestamp in the filesystem is later than that of the source files.

It is interesting to look at it formally, with a rule saying that to derive .o from .c, a function "cc -c" is applied to the source. Make looks at the filesystem for the existence of that .o file, its absence or "out-of-dateness" and, if needed applies the function. If multiple rules are used to derive a sequence of transformations then make will build that chain then execute them.

One interesting question is "how does make build that series of functions, f1..fn, such that:

R = fn(fn-1(fn-2(fn-3..f1(S, L)

I believe it backchains from the redistributes  to build a series of rules which can be applied, then runs those rules forwards to create the final output.

If you view the final redistributables as a set of predicates whose existence is taken as truth, absence as false, and all rules are implies operators the define a path to truth, not falsehood (i.e. we are reasoning over Horn Clauses, then yes, I think you could say "make is a backward chaining system to build a transformation resulting in "truth".

The problem is this: I'm not trying to build something for the sake of building it. I'm trying to build something to ship. I'm running builds and tests repeatedly, from my laptop on my desk, from my laptop in a conference auditorium, a hotel room, and a train doing 190 mph between Brussels and London. A that's all just this week.

Most of the time, those tests have been failing. There are three possible (and non-disjoint) causes of this
  • (a) the null hypothesis: I can't write code that works.
  • (b) a secondary hypothesis: I am better at writing tests to demonstrate the production code is broken than I am at fixing the production code itself.
  • (c) as the external environment changes, so does the outcome of the build process.

Let's pretend that (a) and (b) are false; that I can actually write code that works first time, with the tests intended to show that this condition is not met being well written and correct themselves. Even if such a case held, my build would have been broken for a significant fraction of the time it was this week.

Here's some real examples of fun problems, for "type 3 fun" on the Ordnance Survey Fun Scale.
  1. Build halting as the particular sequence of operations it had chosen depended on maven artifacts which were not only absent, but non-retrievable from a train somewhere under the English Channel.
  2. Connection Reset exceptions talking to an in-VM webapp from within a test case. A test case that worked last week. I never did find the cause of this. Though I eventually concluded that it last worked before I installed a critical OS.X patch (last weeks, not this week's pending one). The obvious action was "reboot the mac' —and lo, it did fix it. I just spent a lot of time on hypotheses (a) and (b) before settling on cause (c)
  3. Tests hanging in the keynote sessions because while my laptop had an internet connection, GET requests against were failing. It turns out that when Jersey starts some servlets up, it tries to do a GET for some XSD file in order to validate some web.xml XML document. If the system is offline, it skips that. But if DNS resolves, then it does a GET and blocks until the document is returned or the GET fails. As as the internet in keynote was a bit overloaded, tests just hung. Fix: edit /etc/hosts to put == or turn off the wifi.
  4. A build at my hotel failing as the run crossed the midnight marker and maven decided to pull down some -SNAPSHOT binaries from mvn central, which were not the binaries I'd just built locally, during the ongoing build and test run.
What do all these have in common? Differences in the environment of the build, primarily networking, except case (4), which was due to the build taking place at a different time from previous builds.

Which brings me to a salient point
The environment of a build includes not only the source files and build rules, it includes the rest of the Internet and the connections to it. furthermore, as the rule engine uses not just the presence/absence of intermediate and final artifacts as triggers for actions, time is an implicit part of the reasoning.
You made could make explict that temporal logic, and have a build tool which look at the timestamp of newly created files in /tmp and flagged up when your filesystem was in a different timezone (Oh, look ant -diagnostics does that! I wonder who wrote that probe?) But it wouldn't be enough, because we've reached a point in which builds are dependent upon state external to even the local machine.

Our builds are therefore, irretrievably nondeterministic.

Categories: FLOSS Project Planets

Bruce Snyder: A Re-Evaluation of My Spinal Cord Injury

Sat, 2015-04-18 00:16
Recently I had to go back at Craig Hospital for a re-evaluation of my spinal cord injury and the news turned out to be pretty good.
Last spring when the accident occurred, I really had no real idea what I was in for with regard to experiencing a spinal cord injury and the very long recovery to follow. And honestly nobody knows what will truly happen with regard to the recovery. The doctors have told me that I have all the good signs of healing and regaining movement, but only time will tell if my body can truly heal itself. 
The re-evaluation consisted of five days worth of inspection by many people looking at your body from every angle. I know it probably doesn't sound like much fun, but after you've already been through everything associated with a spinal cord injury, experiencing a little more inspection isn't a big deal. In fact, I was even eager to hear the results myself because the bottom line is I'm still a patient who relies on the professionals who have helped many others through this devastation many times before. 
The bottom line is that there is still no movement below my knees yet, but I have gotten some feeling back in my feet. My doctor tells me that this is a very good sign, especially in under a year since the accident. I still have a very long way to go but this is definitely good news. At this point in my life, after all I've been through since the accident, I'll take any bit of positive news I can. 
Categories: FLOSS Project Planets

Shawn McKinney: Top 10 Things I Learned at ApacheCon North America 2015

Fri, 2015-04-17 21:21

As another ApacheCon is being wrapped up I wanted to get my thoughts down before they fade.

10. Austin TX is a great city for tech conferences

Where else would the mayor proclaim Apache Software Foundation Days?

Many thanks to the City of Austin @austintexasgovAustin and @MayorSteveAdler for proclaiming 13-16 April "#Apache Software Foundation Days"!

— Apache – The ASF (@TheASF) April 13, 2015

image from:

Situated on the banks of the Colorado river, blessed with an abundance of tech companies, education, the arts (especially music), rich cultural scene, great food, hospitable natives, and a booming economy Austin is an ideal location to host a technology conference.  Kudos to the staff that planned and pulled off this event in flawless fashion.  Y’all did a fantastic job!

9. Apache Software Foundation doesn’t want mean people

Repeated over and over again.  Mean people offend the users, drive off potential project contributors, damage morale, and are not welcome in our communities.

Gina Likins’ keynote Tuesday morning was persuasive.  She taught us politeness is the syntax for effective human conversations and can be learned like programming language syntax.

There are certain words and phrases that trigger stress escalations, e.g. as a matter of fact, actually, and just.

And there is a scale tracking just how offensive a conversation has become:

Defcon Insult Scale for CONversations – DISCON

DISCON Level 1 – Green – Everything is Peachy

DISCON Level 2 – Blue – Non-personal Insults

DISCON Level 3 – Yellow – Added Curse Words

DISCON Level 4 – Orange – Personal Insults

DISCON Level 5 – Red – Threats

There was more, including scientific research into toxicity levels of open source mailing lists.

Check out the YouTube of her keynote here:

How to Thoroughly Insult and Offend People in Open Source

8. JOSE is to REST what WS-Security was to SOAP

You had to know a security topic was going to end up in my list.  :-)

At long last there is a message security facility for REST that’s comparable to ws-security.  Sergey Beryozkin told us all about it:

Practical JOSE with Apache CXF

7. Now that we geeks have inherited the earth, we have a responsibility to be a good ancestor

image from John Field:

Brian Behlendorf’s keynote Monday morning told us why the ASF is still awesome and more, including future objectives (paraphrased below)…

What the ASF Must Do

Increase gender diversity.  We are in a state of crisis in terms of women and technology and it’s getting worse.  We need to figure out a way to make technology workplaces more welcoming to women and minorities.

What the ASF should do

The war over openness is never won.  Many companies today are executing business plans that are attempting to own (or control) layers of technology over the Internet.

There are others who are encroaching on our essential human rights, as they gather information about us and seek to control our experience with technology.

We should guard against these efforts and continue to create software that runs counter to others seeking that control.

We used to joke the geeks will inherit the earth, and now that we have, it’s up to us to be good ancestors.

What the ASF Could Do

The world needs more of us.  The world needs more membership driven, community focused, software development.

Check out his keynote: What’s Still Awesome About the ASF

6. ASF has laid a blueprint

If open source is taking over the world of software, Apache has supplied the blueprint for that takeover.

5. Big Data is here to stay

image from:

The list of Apache projects related to big data, giving talks included Slider, Flink, Kafka, Hadoop, Bigtop, Ignite, Spark, Curnch, and Solr.  These sessions were packed full of technologists scrambling to keep up with the breakneck pace of innovation.

As the corporations and government agencies continue in their attempts to gather and make sense of the piles of data, they need not lose sight of the big picture…

4. Sensible ethics needs to be part of the Big Data Movement

Yes, we can gather and analyze postings from our Twitter, Facebook, Instagram accounts, along with all of the others. But is this the right thing to do?  Sure, it’s profitable and there are advantages to be gained, but are we creating a world in which we want our kids to live? At every step we must ask ourselves, and our governments, these questions.

3. ApacheCon & ASF remain independent of strong commercial interests

I have been attending technology conferences for about 25 years.  Some are paid commercials by their sponsors.

Definitely not so with ApacheCon.  Ross Gardler said it best in his opening keynote:

“They (sponsors) give us money, but they don’t expect anything in return.   We give them a thank you page and that is it.  No influence of the projects, or even the foundation.”

See more in his State of the Feather YouTube video.

2. TEALS is the best thing to happen inside American schools in a long time

Kevin Wang of Microsoft gave a highly inspirational talk:

Volunteer to Teach High School CS Without Quitting Your Day Job

In it, he describes a program, sponsored by Microsoft, that recruits and trains volunteers from the professional ranks to mentor high school teachers in computer science.  The costs are minimal for the schools, around $100 per student per course.  It introduces computer science curriculum to kids and increases their chances of continuing on to college later on.

The corporate promise is the ability to meet projected future needs with local talent.

The human promise is that more kids, including girls, minorities, and financially disadvantaged, be offered new opportunities in their lives, through the study of computer science in high school.

It’s a win-win scenario.

More info on their website: TEALS (Technology Education And Literacy in Schools) 1.  I am trying to love Microsoft (really)

Those who know me well realize how difficult it is to say; after decades of anti-competitive practices and the promotion of closed technologies, they are starting to come around.

Jay Schmelzer’s keynote has (almost) convinced me that his company, behind the leadership of a new CEO has turned over a new leaf.  Microsoft is promoting open systems and open standards.  Hurray!!!

Hear more in his keynote: 20 Years of Valuable Lessons for the World

Categories: FLOSS Project Planets

Justin Mason: Links for 2015-04-17

Fri, 2015-04-17 18:58
Categories: FLOSS Project Planets

Jeremy Quinn: Ascension [Flickr]

Fri, 2015-04-17 06:37

sharkbait posted a photo:

Stairway to Canning Town.
On my way to the Woolwich foot tunnel, to start section1 of the Capital Ring.

Categories: FLOSS Project Planets

Jeremy Quinn: Keep Left No Cycling [Flickr]

Fri, 2015-04-17 06:26

sharkbait posted a photo:

I am wary of rogue cyclists at the moment, I got knocked down by one a while ago.

Categories: FLOSS Project Planets

Justin Mason: Links for 2015-04-16

Thu, 2015-04-16 18:58
  • Extracting Structured Data From Recipes Using Conditional Random Fields

    nice probabilistic/ML approach to recipe parsing

    (tags: nytimes recipes parsing text nlp machine-learning probabilistic crf++ algorithms feature-extraction)

  • Large-scale cluster management at Google with Borg

    Google’s Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines. It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification language, name service integration, real-time job monitoring, and tools to analyze and simulate system behavior. We present a summary of the Borg system architecture and features, important design decisions, a quantitative analysis of some of its policy decisions, and a qualitative examination of lessons learned from a decade of operational experience with it. (via Conall)

    (tags: via:conall clustering google papers scale to-read borg cluster-management deployment packing reliability redundancy)

  • Keeping Your Car Safe From Electronic Thieves –

    In a normal scenario, when you walk up to a car with a keyless entry and try the door handle, the car wirelessly calls out for your key so you don’t have to press any buttons to get inside. If the key calls back, the door unlocks. But the keyless system is capable of searching for a key only within a couple of feet. Mr. Danev said that when the teenage girl turned on her device, it amplified the distance that the car can search, which then allowed my car to talk to my key, which happened to be sitting about 50 feet away, on the kitchen counter. And just like that, open sesame. What the hell — who designed a system that would auto-unlock based on signal strength alone?!!

    (tags: security fail cars keys signal proximity keyless-entry prius toyota crime amplification power-amplifiers 3db keyless)

  • Closed access means people die

    ‘We’ve paid 100 BILLION USD over the last 10 years to “publish” science and medicine. Ebola is a massive systems failure.’ See also : ‘The conventional wisdom among public health authorities is that the Ebola virus, which killed at least 10,000 people in Liberia, Sierra Leone and Guinea, was a new phenomenon, not seen in West Africa before 2013. [...] But, as the team discovered, that “conventional wisdom” was wrong. In fact, they found a bunch of studies, buried behind research paywalls, that revealed that there was significant evidence of antibodies to the Ebola virus in Liberia and in other nearby nations. There was one from 1982 that noted: “medical personnel in Liberian health centers should be aware of the possibility that they may come across active cases and thus be prepared to avoid nosocomial epidemics.”

    (tags: deaths liberia ebola open-access papers elsevier science medicine reprints)

  • Making Pinterest — Learn to stop using shiny new things and love MySQL

    ‘The third reason people go for shiny is because older tech isn’t advertised as aggressively as newer tech. The younger companies needs to differentiate from the old guard and be bolder, more passionate and promise to fulfill your wildest dreams. But most new tech sales pitches aren’t generally forthright about their many failure modes. In our early days, we fell into this third trap. We had a lot of growing pains as we scaled the architecture. The most vocal and excited database companies kept coming to us saying they’d solve all of our scalability problems. But nobody told us of the virtues of MySQL, probably because MySQL just works, and people know about it.’ It’s true! — I’m still a happy MySQL user for some use cases, particularly read-mostly relational configuration data…

    (tags: mysql storage databases reliability pinterest architecture)

  • Microservices and elastic resource pools with Amazon EC2 Container Service

    interesting approach to working around ECS’ shortcomings — bit specific to Hailo’s microservices arch and IPC mechanism though. aside: I like their version numbering scheme: ISO-8601, YYYYMMDDHHMMSS. keep it simple!

    (tags: versioning microservices hailo aws ec2 ecs docker containers scheduling allocation deployment provisioning qos)

  • Please Kill Me (Eventually) | Motherboard

    There is much that the wise application of technology can do to help us ease off this mortal coil, instead of tormenting ourselves at the natural end of life in a futile, undignified and excruciating attempt to keep it somehow duct-taped on. Train more people in geriatrics, for example. Learn new ways to make life safe, healthy, fun and interesting for the old. Think like a community, a brotherhood, not like atomized competing individuals a few of whom can somehow “beat the system” of the universe. Maybe it is better to examine clearly what we are with a view to understanding and acceptance than it is to try to escape what perhaps should be our inevitable ending.

    (tags: death mortality cryogenics alcor geriatrics life singularity mind-uploading ray-kurzweil)

  • CGA in 1024 Colors – a New Mode: the Illustrated Guide

    awesome hackery. brings me back to my C=64 demo days

    (tags: pc cga graphics hacks art 1024-colours)

Categories: FLOSS Project Planets

Jeremy Quinn: Aground [Flickr]

Thu, 2015-04-16 05:21

sharkbait posted a photo:

A derelict ferry, aground at low tide, just downstream of the Thames Barrier.

Categories: FLOSS Project Planets