Apache Spark is an open source cluster computing framework. In contrast to Hadoop’s two-stage disk-based MapReduce paradigm, Spark’s in-memory primitives provide performance up to 100 times faster for certain applications.
Come on, admit it.
We just don't have a freaking clue.
- Delayed flash crash arrest may herald future spoofing detection woesTim Massad, the head of the Commodity Futures Trading Commission, which oversees the trading of futures and swaps, said on Wednesday that it took so long to charge Sarao because of the size and complexity of U.S. derivatives markets. "These are huge markets," he said. "There's a lot going on."
- Trading at the speed of lightThe faster your trading system, the quicker you can take advantage of those discrepancies. But that is not all the high-frequency traders can do. They can “front-run” news events, jumping on anything that might move a price, and get in and out of a stock before anyone else knows that something has happened. By trading at such speed, they can even get into an equity in the gap between the placing of a large buy order and the execution of it, and thereby buy just before the price rises.
- ‘Flash crash’ charges spark alarm over regulation of US marketsSherrod Brown, the top Democrat on the Senate banking committee, told the Financial Times: “It’s encouraging that the Justice Department and [Commodity Futures Trading Commission] are pursuing this case, but troubling that it has only come to light now with the help of a whistleblower who invested substantial time in putting the pieces together.”
- 'Flash Crash' arrest shakes investors' confidenceAdding to concerns, the CFTC was alerted to Sarao's alleged misdeeds by a whistle-blower, who has not been identified, according to Shayne Stevenson, who represents the whistle-blower through Hagens Berman law firm in Seattle. Stevenson said his client brought "high-quality information" about "market manipulation" to the CFTC, which alerted the DOJ.
- Accused British 'flash crash' trader fights extradition to U.S.A British man accused of market manipulation that contributed to the May 2010 Wall Street "flash crash" said he opposed being extradited to the United States, while the operator of the market where he traded sought to rebut prosecutors' suggestion that futures helped cause the crash.
- Roots of 'flash crash' go back further than you thought at CMEMore than a year before the May 6, 2010 "flash crash," CME Group noticed questionable trading in its E-mini market by a particular electronic trader who was placing orders and cancelling them.
As the crash whipsawed the futures and stock markets in 2010, CME saw the suspicious activity again and warned the trader that day that orders must be placed “in good faith,” without an intent to cancel. The trader responded two weeks later: “Kiss my ass.”
- How computerized trading in the hands of a nobody in Britain allegedly crashed the stock marketNot everyone agrees.
- A Sweatpants-Wearing Rando Might Have Caused the Flash CrashHe seems to have executed his trades out of a modest, semi-detached house under the Heathrow flight path that he shared with his parents. He used off-the-shelf software that he souped up to make his bets. He named one shell company "Nav Sarao Milking Markets Ltd." The guy showed up to court in a pair of white sweatpants.
- 'Flash Crash' Arrest Raises More Questions Than AnswersBut surely, since then regulators went through the action that took place on and around the day of the Flash Crash with a very fine comb? And if so, why did Sarao's alleged actions not jump at them as strange at the time?
- Why Did It Take So Long To Catch The ‘Flash Crash’ Futures Trader?The CME Group declined to comment, citing the ongoing investigation. But analysts worry that the CME’s revenue model interferes with its motivation to police trading. The more trades that zip through the exchange, the more money it makes. That means it could be disincentivized from tackling manipulative traders who still bring valuable liquidity to the market.
- Spoofing the Stock Market -- Here’s One Way to Stop ItIt may be a bit more complicated than that, but not much.
- Junk bonds really to blame for Flash CrashJunk Debt collapsed a full hour before the stock market Flash Crash took place (and I would add that utilities and Treasurys were leading prior, indicating a VIX spike was possible). If junk debt collapses, how can stocks not collapse afterward when in a bankruptcy proceeding, junk debt has a higher claim on assets than equity? What caused the Flash Crash in stocks was a Flash Crash in junk debt, which was like a 1987 style crash in credit.
It's not like this is anything important.
It's not like it's the entire world economy.
Who cares whether we understand it or not?
Good slide deck from Etsy’s Mike Brittain regarding their CD setup. Some interesting little-known details: Slide 41: database schema changes are not CD’d — they go out on “Schema change Thursdays”. Slide 44: only the webapp is CD’d — PHP, Apache, memcache components (Etsy.com, support and back-office tools, developer API, gearman async worker queues). The external “services” are not — databases, Solr/JVM search (rolling restarts), photo storage (filters, proxy cache, S3), payments (PCI-DSS, controlled access). They avoid schema changes and breaking changes using an approach they call “non-breaking expansions” — expose new version in a service interface; support multiple versions in the consumer. Example from slides 50-63, based around a database schema migration. Slide 66: “dev flags” (rollout oriented) are promoted to “feature flags” (long lived degradation control). Slide 71: some architectural philosophies: deploying is cheap; releasing is cheap; gathering data should be cheap too; treat first iterations as experiments. Slide 102: “Canary pools”. They have multiple pools of users for testing in production — the staff pool, users who have opted in to see prototypes/beta stuff, 0-100% gradual phased rollout.
Good info on how Etsy use their Deployinator tool, end-to-end. Slide 11: git SHA is visible for each env, allowing easy verification of what code is deployed. Slide 14: Code is deployed to “princess” staging env while CI tests are running; no need to wait for unit/CI tests to complete. Slide 23: smoke tests of pre-prod “princess” (complete after 8 mins elapsed). Slide 31: dashboard link for deployed code is posted during deploy; post-release prod smoke tests are run by Jenkins. (short ones! they complete in 42 seconds)
good aspirational checklist, inspired heavily by James Hamilton’s seminal 2007 paper, “On Designing And Deploying Internet-Scale Services”
In particular, the CFTC release notes:in or about June 2009, Defendants modified a commonly used off-the-shelf trading platform to automatically simultaneously “layer” four to six exceptionally large sell orders into the visible E-mini S&P central limit order book (the Layering Algorithm), with each sell order one price level from the other. As the E-mini S&P futures price moved, the Layering Algorithm allegedly modified the price of the sell orders to ensure that they remained at least three or four price levels from the best asking price; thus, remaining visible to other traders, but staying safely away from the best asking price. Eventually, the vast majority of the Layering Algorithm orders were canceled without resulting in any transactions. According to the Complaint, between April 2010 and April 2015, Defendants utilized the Layering Algorithm on over 400 trading days.
The Complaint alleges that Defendants often cycled the Layering Algorithm on and off several times during a typical trading day to create large imbalances in the E-mini S&P visible order book to affect the prevailing E-mini S&P price. Defendants then allegedly traded in a manner designed to profit from this temporary artificial volatility. According to the Complaint, from April 2010 to present, Defendants have profited over $40 million, in total, from E-mini S&P trading.
As others quickly pointed out, the notion that "layering" is involved in these wild price swings is being studied by multiple agencies. For example: Exclusive: SEC targets 10 firms in high frequency trading probe - SEC document.The SEC has been seeking evidence of abuse of order types, as well as traditional forms of abusive trading like "layering" or "spoofing" and other issues relating to high-frequency trading that might be violations of the law, SEC Director of Enforcement Andrew Ceresney told Reuters in May (reut.rs/1kwSqF5).
Spoofing and layering are tactics where traders places orders that they cancel before they are executed to create the false impression of demand, aiming to trick others into buying or selling a stock at the artificial price.
I'm pleased that investigators continue to investigate.
On the other hand, even after 5 years the investigators still appear to be uncertain as to exactly what happened and why.
It's disturbing news, all around.
Wow, this is staggering.The Justice Department and FBI have formally acknowledged that nearly every examiner in an elite FBI forensic unit gave flawed testimony in almost all trials in which they offered evidence against criminal defendants over more than a two-decade period before 2000. [....] The review confirmed that FBI experts systematically testified to the near-certainty of “matches” of crime-scene hairs to defendants, backing their claims by citing incomplete or misleading statistics drawn from their case work. In reality, there is no accepted research on how often hair from different people may appear the same. Since 2000, the lab has used visual hair comparison to rule out someone as a possible source of hair or in combination with more accurate DNA testing. Warnings about the problem have been mounting. In 2002, the FBI reported that its own DNA testing found that examiners reported false hair matches more than 11 percent of the time.
Using Steam streaming and EC2 g2.2xlarge spot instances — ‘comes out to around $0.52/hr’. That’s pretty compelling IMO
actually an officially-supported mode. huh
The below is an abstract for a talk...Open Source Has Won The Battle; Let's Not Lose The War
20 years ago, a bunch of us got together and created Apache, and then 5 years later went ahead and created The Apache Software Foundation. The idea of Open Source back then was weird, and wild, and suspect. But due to the power and capability of the Apache Web Server, in combination with Linux, Open Source gained traction, acceptance and now ubiquity.
Looking around at the IT landscape nowadays, Open Source is found everywhere. Software is eating the world, and Open Source is the utensil of choice. Corporations once critical of Open Source, now embrace it; Open Source is now both strategic and mandatory. In many ways, one could assume that Open Source has won.
Well, maybe it has won, but it's just won the battle; the war is still there, and our success in winning the battle is threatening to cause our loss of the war.
"It's on Github, so of course it's Open Source, right?" Wrong.
"It's got an OSI license, so nothing else is needed, right?" Wrong.
"There's nothing wrong with paid developers/contributors, right?" Well... maybe yes and maybe no.
"What is really the matter with pay-to-play Open Source foundations?" Give me 30 minutes or so, and I'll tell you what the risks are.
There's an old saying that in Open Source, developers/contributors scratch their own itches. But what about today? Do they still? Can they still? And what is the ultimate harm if they can't. And as more and more Open Source gets funded, directly, by corporations, where does that leave the true volunteer contributor? And finally, who really has the ultimate control over a project's destiny?
This presentation will give a short history of the Open Source movement, and why the most critical forces behind its success in being an innovation engine may be at risk.
Young stargazers, Lottie and MillieIn the morning, we found ourselves in a planetarium tent, then we looked at sunspots, and held pieces of meteorite.
The evening event was stargazing at Holy Trinity Church in Hardwicke, just outside Hay. Quite apart from the lack of light pollution, the location was a special one, since the vicar of the parish from 1856 until 1885 was Reverend Thomas William Webb, who in his spare time observed the night sky with telescopes and an observatory he had built himself.
Holy Trinity Church, Hardwicke
In 1859, while at Hardwicke he wrote the classic book, Celestial Objects for the Common Telescope, the object of which was "to furnish the possessors of ordinary telescopes with plain directions for their use, and a list of objects for their advantageous employment".
The book remained in print well into the following century (and was recently republished by Cambridge University Press), and it's probably difficult to overemphasise the importance of this book in encouraging generation after generation of amateur stargazers.
In the words of Janet and Mark Robinson, who used to live in the vicarage and have edited a book about Webb,
Like Patrick Moore, he was an enthusiast who wanted to inspire as many people as possible to look through a telescope. Even at the choir party he "arranged the telescope and acted as showman and all in turn had a look at Saturn".Webb would no doubt have been pleased to see yesterday's gathering of enthusiastic amateurs (including the Robinsons) with an impressive range of telescopes, on a cold but very clear night. The highlight for us was seeing Jupiter and its four brightest moons (Io, Europa, Ganymede and Callisto) through a large reflecting telescope. We could even see the north and south belts, and the Great Red Spot (or Pink Splodge as Lottie named it).
Sunset. Venus is visible top centreThank you to the organisers of the Hay Dark Skies Festival, and the volunteers from the Usk Astronomical Society (the oldest astronomical society in the UK), the Abergavenny Astronomy Society and the Heads of the Valleys Astronomical Society.
wow, Walter Higgins’ book (from Peachpit Press) is looking great
Pinterest’s Hadoop workflow manager; ‘scalable, reliable, simple, extensible’ apparently. Hopefully it allows upgrades of a workflow component without breaking an existing run in progress, like LinkedIn’s Azkaban does :(
Boy, my mind has really been wandering recently.
Must be the weather.
- The Town That Creep Built“Fault creep” is a condition that results when the underlying geology is too soft to get stuck or to accumulate tectonic stress: in other words, the deep rocks beneath Hollister are slippery, more pliable, and behave a bit like talc. Wonderfully but unsurprisingly, the mechanism used to study creep is called a creepmeter.
The ground sort of oozes past itself, in other words, a slow-motion landslide at a pace that would be all but imperceptible if it weren’t for the gridded streets and property lines being bent out of shape above it.
- SevenevesThe moon blew up without warning and for no apparent reason. It was waxing, only one day short of full. The time was 05:03:12 UTC. Later it would be designated A+0.0.0, or simply Zero.
- Linux 4.0But "Hurr durr I'ma sheep" trounced "I like online polls" by a 62-to-38% margin, in a poll that people weren't even supposed to participate in. Who can argue with solid numbers like that? 5,796 votes from people who can't even follow the most basic directions?
- Linux 4.0 releasedLooking at just the statistics in git, this release is not just when we cross half a million commits total, but also cross the 4 million git object limit. Interestingly (if you look for numeric patterns), Linux 3.0 was when we crossed a quarter million commits and 2 million git objects, so there's a nice (and completely unintentional) pattern there when it comes to the kernel git repository.
- Almonds in California: They use up a lot of water, but they deserve a place in California's future.The problem is that, thanks to the current drought, the water supply is going away faster than expected. The almond industry is an indicator of how difficult it might be to adapt to climate change, economically and environmentally.
What we’re witnessing in California right now is a glimpse into the future. California has now endured drought in 11 of the last 15 years, and there’s every reason to believe this is just the beginning.
- Math for eight-year-olds: graph theory for kids!This morning I had the pleasure to be a mathematical guest in my daughter’s third-grade class, full of inquisitive eight- and nine-year-old girls, and we had a wonderful interaction. Following up on my visit last year (math for seven-year-olds), I wanted to explore with them some elementary ideas in graph theory, which I view as mathematically rich, yet accessible to children.
- Replicating SQLite using Raft ConsensusI decided to build a distributed replication layer using the Raft consensus protocol, which gives me effective replication without the hassle of running a much heavier solution like MySQL. It provides all the advantages of replication, with the data modelling functionality of a relational database, but with the convenience of a single-file database.
- A Million Little BoxesWhile crossword construction is unavoidably linguistic, constructors these days rely heavily on data and programming. Take, for instance, Puzzle 5, always the hardest of the tournament. This year, Jeff Chen was the sadist behind the curtain. An aspiring novelist from Seattle, he runs the crossword database XWord Info and has authored or co-authored 37 New York Times puzzles.
Chen’s Puzzle 5 was titled “Attention, Newbies!” The conceit was adding new B’s (get it?) to familiar phrases. “Vocalist” became “vocab list,” “alloy” became “ballboy,” and so on. To find workable and interesting phrases like this, Chen wrote a Python script and applied it to his master word and phrase list. Using this program, he realized “caroms” could become “car bombs,” for example.
- From the vault: Watching (and re-watching) “The Mother of All Demos”To give an idea of the scope of the demo, Engelbart demonstrated an early look at word processing, windowing, hypertext, and dynamic file linking, as well as using graphics in a computer program. It was also the first time many of the attendees had seen a mouse, although work on the mouse began in 1963.
- Arcology: cutaways of the future city-hives that never wereSoleri taught architecture at ASU, but his real classroom was Arcosanti: "an experimental town and molten bronze bell casting community" a few hours from Phoenix, which was supposed to have 5,000 residents by now, demonstrating "lean" city living in high-density complexes that combined super-efficient usage of space with stylized, naturalistic exteriors that made each building part of the landscape.
- The Cult of Work You Never Meant to JoinI take walks. I leave my phone in my pocket when I’m out with friends or eating my meals. I spend a fair amount of time on my hobbies, like writing and hunting for the world’s best cheeseburger.
- Code QualityIt's like a salad recipe written by a corporate lawyer using a phone auto-correct that only knew Excel formulas.
It’s an item that we’re very proud of, and of some historical significance.
The conference producers treated it like it was the Declaration of Independence or something. They handled it carefully and reverently.
At the end of the event the guy in charge of A/V came to me with some twine.
He said he had removed it from the hanging hooks on the feather in order to use black nylon that matched the stage dressing, and which would hang more securely. But he saved these scraps of twine because he knew how significant the item was to us.
Now, it’s not that the twine mattered – it was something I added years after the original was made. It’s that he cared enough, and respected our heritage enough to save it and track me down, that impressed me so very much. It really put a wonderful final touch on an almost-perfect event.
And this is why, among many other reasons, we love our conference production company, The Linux Foundation.
Someone has put up a thoughtful post on whether you can view make as a proof system. I thought about it and have come up with a conclusion: maybe in the past —but not any more.
As background, it's worth remembering
- I did "write the book on Ant", which is a build system in which you declare the transformational functions required to get your application into a state which you consider shippable/deployable, a set of functions which you define in a DAG for the Ant execution engine to generate and order from and then apply to your environment
- Early last year, in my spare time, I formally specified the Hadoop FS APIs in Spivey's Z notation, albeit in a Python syntax.
- In YARN-913 I went on to specify the desired behaviour of the YARN registry as a TLA+ specification.
This means that (a) I believe I understand about specifications, (b) I vaguely remember what a proof engine is, and (b) I understand how Prolog resolves things, and specifically, why "!" is the most useful operator when you are trying to write applications.
Now I note that the author of that first post, Bob Atkey, does not only has a background of formal logic, SML and possibly even worked with the same people as me, his knowledge is both greater and more up to date than mine. I just have more experience of breaking builds and getting emails from jenkins telling me this.
Now, consider a software project's build
- A set of source artifacts, S, containing artifacts s1..sn
- A declaration of the build process, B
- A set of external dependencies, libraries, L.
- A build environment, E., comprising the tools needed for the build, and the computer environment around them, including the filesystem, OS, etc.
The purpose of the build system, then, is to generate R from S through a series of functions applied to (S, L) with tools T within the environment E. The build process description, B, defines or declares that process.
There's many ways to do that; a single line bash file cc *.c && cc *.o could be enough to compile the lexical analyser example from Ch02 of the dragon book.
Builds are more complex than that, which is where tools like make come in.
Make essentially declares that final list of redistributables, R, and a set of transformations from inputs artifacts to output artifacts, including the functions (actions by the tools) to generate the output artifacts from the input artifacts.
The make runtime looks at what artifacts exist, what are missing, and what are out of date somehow builds a chain of operations that are hypothesised to produce a complete set of output artifact whose timestamp in the filesystem is later than that of the source files.
It is interesting to look at it formally, with a rule saying that to derive .o from .c, a function "cc -c" is applied to the source. Make looks at the filesystem for the existence of that .o file, its absence or "out-of-dateness" and, if needed applies the function. If multiple rules are used to derive a sequence of transformations then make will build that chain then execute them.
One interesting question is "how does make build that series of functions, f1..fn, such that:
R = fn(fn-1(fn-2(fn-3..f1(S, L)
I believe it backchains from the redistributes to build a series of rules which can be applied, then runs those rules forwards to create the final output.
If you view the final redistributables as a set of predicates whose existence is taken as truth, absence as false, and all rules are implies operators the define a path to truth, not falsehood (i.e. we are reasoning over Horn Clauses, then yes, I think you could say "make is a backward chaining system to build a transformation resulting in "truth".
The problem is this: I'm not trying to build something for the sake of building it. I'm trying to build something to ship. I'm running builds and tests repeatedly, from my laptop on my desk, from my laptop in a conference auditorium, a hotel room, and a train doing 190 mph between Brussels and London. A that's all just this week.
Most of the time, those tests have been failing. There are three possible (and non-disjoint) causes of this
- (a) the null hypothesis: I can't write code that works.
- (b) a secondary hypothesis: I am better at writing tests to demonstrate the production code is broken than I am at fixing the production code itself.
- (c) as the external environment changes, so does the outcome of the build process.
Let's pretend that (a) and (b) are false; that I can actually write code that works first time, with the tests intended to show that this condition is not met being well written and correct themselves. Even if such a case held, my build would have been broken for a significant fraction of the time it was this week.
Here's some real examples of fun problems, for "type 3 fun" on the Ordnance Survey Fun Scale.
- Build halting as the particular sequence of operations it had chosen depended on maven artifacts which were not only absent, but non-retrievable from a train somewhere under the English Channel.
- Connection Reset exceptions talking to an in-VM webapp from within a test case. A test case that worked last week. I never did find the cause of this. Though I eventually concluded that it last worked before I installed a critical OS.X patch (last weeks, not this week's pending one). The obvious action was "reboot the mac' —and lo, it did fix it. I just spent a lot of time on hypotheses (a) and (b) before settling on cause (c)
- Tests hanging in the keynote sessions because while my laptop had an internet connection, GET requests against java.sun.com were failing. It turns out that when Jersey starts some servlets up, it tries to do a GET for some XSD file in order to validate some web.xml XML document. If the system is offline, it skips that. But if DNS resolves java.sun.com, then it does a GET and blocks until the document is returned or the GET fails. As as the internet in keynote was a bit overloaded, tests just hung. Fix: edit /etc/hosts to put java.sun.com == 127.0.0.1 or turn off the wifi.
- A build at my hotel failing as the run crossed the midnight marker and maven decided to pull down some -SNAPSHOT binaries from mvn central, which were not the binaries I'd just built locally, during the ongoing build and test run.
Which brings me to a salient point
The environment of a build includes not only the source files and build rules, it includes the rest of the Internet and the connections to it. furthermore, as the rule engine uses not just the presence/absence of intermediate and final artifacts as triggers for actions, time is an implicit part of the reasoning.
You made could make explict that temporal logic, and have a build tool which look at the timestamp of newly created files in /tmp and flagged up when your filesystem was in a different timezone (Oh, look ant -diagnostics does that! I wonder who wrote that probe?) But it wouldn't be enough, because we've reached a point in which builds are dependent upon state external to even the local machine.
Our builds are therefore, irretrievably nondeterministic.
Last spring when the accident occurred, I really had no real idea what I was in for with regard to experiencing a spinal cord injury and the very long recovery to follow. And honestly nobody knows what will truly happen with regard to the recovery. The doctors have told me that I have all the good signs of healing and regaining movement, but only time will tell if my body can truly heal itself.
The re-evaluation consisted of five days worth of inspection by many people looking at your body from every angle. I know it probably doesn't sound like much fun, but after you've already been through everything associated with a spinal cord injury, experiencing a little more inspection isn't a big deal. In fact, I was even eager to hear the results myself because the bottom line is I'm still a patient who relies on the professionals who have helped many others through this devastation many times before.
The bottom line is that there is still no movement below my knees yet, but I have gotten some feeling back in my feet. My doctor tells me that this is a very good sign, especially in under a year since the accident. I still have a very long way to go but this is definitely good news. At this point in my life, after all I've been through since the accident, I'll take any bit of positive news I can.
As another ApacheCon is being wrapped up I wanted to get my thoughts down before they fade.10. Austin TX is a great city for tech conferences
Where else would the mayor proclaim Apache Software Foundation Days?
— Apache – The ASF (@TheASF) April 13, 2015
Situated on the banks of the Colorado river, blessed with an abundance of tech companies, education, the arts (especially music), rich cultural scene, great food, hospitable natives, and a booming economy Austin is an ideal location to host a technology conference. Kudos to the staff that planned and pulled off this event in flawless fashion. Y’all did a fantastic job!9. Apache Software Foundation doesn’t want mean people
Repeated over and over again. Mean people offend the users, drive off potential project contributors, damage morale, and are not welcome in our communities.
Gina Likins’ keynote Tuesday morning was persuasive. She taught us politeness is the syntax for effective human conversations and can be learned like programming language syntax.
There are certain words and phrases that trigger stress escalations, e.g. as a matter of fact, actually, and just.
And there is a scale tracking just how offensive a conversation has become:Defcon Insult Scale for CONversations – DISCON
DISCON Level 1 – Green – Everything is Peachy
DISCON Level 2 – Blue – Non-personal Insults
DISCON Level 3 – Yellow – Added Curse Words
DISCON Level 4 – Orange – Personal Insults
DISCON Level 5 – Red – Threats
There was more, including scientific research into toxicity levels of open source mailing lists.
Check out the YouTube of her keynote here:JOSE is to REST what WS-Security was to SOAP
You had to know a security topic was going to end up in my list. :-)
At long last there is a message security facility for REST that’s comparable to ws-security. Sergey Beryozkin told us all about it:
Brian Behlendorf’s keynote Monday morning told us why the ASF is still awesome and more, including future objectives (paraphrased below)…What the ASF Must Do
Increase gender diversity. We are in a state of crisis in terms of women and technology and it’s getting worse. We need to figure out a way to make technology workplaces more welcoming to women and minorities.What the ASF should do
The war over openness is never won. Many companies today are executing business plans that are attempting to own (or control) layers of technology over the Internet.
There are others who are encroaching on our essential human rights, as they gather information about us and seek to control our experience with technology.
We should guard against these efforts and continue to create software that runs counter to others seeking that control.
We used to joke the geeks will inherit the earth, and now that we have, it’s up to us to be good ancestors.What the ASF Could Do
The world needs more of us. The world needs more membership driven, community focused, software development.
Check out his keynote: What’s Still Awesome About the ASF6. ASF has laid a blueprint
If open source is taking over the world of software, Apache has supplied the blueprint for that takeover.5. Big Data is here to stay
The list of Apache projects related to big data, giving talks included Slider, Flink, Kafka, Hadoop, Bigtop, Ignite, Spark, Curnch, and Solr. These sessions were packed full of technologists scrambling to keep up with the breakneck pace of innovation.
As the corporations and government agencies continue in their attempts to gather and make sense of the piles of data, they need not lose sight of the big picture…4. Sensible ethics needs to be part of the Big Data Movement
Yes, we can gather and analyze postings from our Twitter, Facebook, Instagram accounts, along with all of the others. But is this the right thing to do? Sure, it’s profitable and there are advantages to be gained, but are we creating a world in which we want our kids to live? At every step we must ask ourselves, and our governments, these questions.3. ApacheCon & ASF remain independent of strong commercial interests
I have been attending technology conferences for about 25 years. Some are paid commercials by their sponsors.
Definitely not so with ApacheCon. Ross Gardler said it best in his opening keynote:
“They (sponsors) give us money, but they don’t expect anything in return. We give them a thank you page and that is it. No influence of the projects, or even the foundation.”
See more in his State of the Feather YouTube video.2. TEALS is the best thing to happen inside American schools in a long time
Kevin Wang of Microsoft gave a highly inspirational talk:
In it, he describes a program, sponsored by Microsoft, that recruits and trains volunteers from the professional ranks to mentor high school teachers in computer science. The costs are minimal for the schools, around $100 per student per course. It introduces computer science curriculum to kids and increases their chances of continuing on to college later on.
The corporate promise is the ability to meet projected future needs with local talent.
The human promise is that more kids, including girls, minorities, and financially disadvantaged, be offered new opportunities in their lives, through the study of computer science in high school.
It’s a win-win scenario.More info on their website: TEALS (Technology Education And Literacy in Schools) 1. I am trying to love Microsoft (really)
Those who know me well realize how difficult it is to say; after decades of anti-competitive practices and the promotion of closed technologies, they are starting to come around.
Jay Schmelzer’s keynote has (almost) convinced me that his company, behind the leadership of a new CEO has turned over a new leaf. Microsoft is promoting open systems and open standards. Hurray!!!
Hear more in his keynote: 20 Years of Valuable Lessons for the World
some basic ECS tips from Gilt
Essentially, add a central GC scheduler to improve tail latencies in a cluster, by taking instances out of the pool to perform slow GC activity instead of letting them impact live operations. I’ve been toying with this idea for a while, nice to see a solid paper about it
Nice, simple “build a website” platform. Keeping this one bookmarked for the next time someone non-techie asks me for the simplest way to do just that (thanks for the tip, Oisin)
sharkbait posted a photo:
Stairway to Canning Town.
On my way to the Woolwich foot tunnel, to start section1 of the Capital Ring.
sharkbait posted a photo:
I am wary of rogue cyclists at the moment, I got knocked down by one a while ago.
nice probabilistic/ML approach to recipe parsing
‘We’ve paid 100 BILLION USD over the last 10 years to “publish” science and medicine. Ebola is a massive systems failure.’ See also https://www.techdirt.com/articles/20150409/17514230608/dont-think-open-access-is-important-it-might-have-prevented-much-ebola-outbreak.shtml : ‘The conventional wisdom among public health authorities is that the Ebola virus, which killed at least 10,000 people in Liberia, Sierra Leone and Guinea, was a new phenomenon, not seen in West Africa before 2013. [...] But, as the team discovered, that “conventional wisdom” was wrong. In fact, they found a bunch of studies, buried behind research paywalls, that revealed that there was significant evidence of antibodies to the Ebola virus in Liberia and in other nearby nations. There was one from 1982 that noted: “medical personnel in Liberian health centers should be aware of the possibility that they may come across active cases and thus be prepared to avoid nosocomial epidemics.”
‘The third reason people go for shiny is because older tech isn’t advertised as aggressively as newer tech. The younger companies needs to differentiate from the old guard and be bolder, more passionate and promise to fulfill your wildest dreams. But most new tech sales pitches aren’t generally forthright about their many failure modes. In our early days, we fell into this third trap. We had a lot of growing pains as we scaled the architecture. The most vocal and excited database companies kept coming to us saying they’d solve all of our scalability problems. But nobody told us of the virtues of MySQL, probably because MySQL just works, and people know about it.’ It’s true! — I’m still a happy MySQL user for some use cases, particularly read-mostly relational configuration data…
interesting approach to working around ECS’ shortcomings — bit specific to Hailo’s microservices arch and IPC mechanism though. aside: I like their version numbering scheme: ISO-8601, YYYYMMDDHHMMSS. keep it simple!
awesome hackery. brings me back to my C=64 demo days
sharkbait posted a photo:
A derelict ferry, aground at low tide, just downstream of the Thames Barrier.