Planet Apache

Syndicate content
Updated: 4 hours 51 min ago

Chiradeep Vittal: How did they build that — EC2 Enhanced Networking

Fri, 2014-11-21 01:45

Among the flurry of new features introduced by AWS in 2013, is a performance enhancement known as ‘Enhanced Networking‘. According to the blurb: ” enhanced networking on your instance results in higher performance (packets per second), lower latency, and lower jitter’. The requirements are that you install an Intel 10GbE driver (ixgbef) in your instance and enable a feature called SR-IOV.

The AWS cloud is built around virtualization technology — specifically your instances are virtual machines running on top of a version of the open source Xen Hypervisor.
The hypervisor is what guarantees the isolation between my instance and your instance when they both run on the same set of CPUs.

The hypervisor intercepts all I/O from the virtual machine so that the virtual machine is abstracted from the hardware — this provides security as well as portability since the VM doesn’t need to care about the drivers for the I/O hardware. The VM sees a NIC that is software defined and as a result the hypervisor can inspect all traffic to and from the VM. -This allows AWS to control the networking traffic between the VM and the rest of the infrastructure. This is used to deliver features such as security groups and ACL.

The downside of processing all network traffic to/from the VM is that the host CPU cycles are consumed processing this traffic. This is quite a significant overhead compared to a bare-metal instance. The hypervisor needs to apply stateful firewall rules on every packet, switch the packet and encapsulate it. Some estimates put this overhead as high as 70% of the CPU available to the hypervisor (at 10 Gb/s). Software processing also introduces problems of noisy neighbors — variable jitter and high latency at 10Gbps are common.

Fortunately, SR-IOV (Single Root IO Virtualization) provides a direct path for the VM to access the underlying hardware NIC. Bypassing the hypervisor leads to line-rate performance. Enhanced Networking takes advantage of this: in order to benefit from this, your AMI needs to have SR-IOV drivers installed in it.

Great — but now that the hypervisor is out of the path, how does AWS provide software-defined features such as security groups and ACL? The current generation of SR-IOV NICs (AWS uses the Intel 82599 ) do not have stateful firewalls or the ability to have process large number of ACL. Furthermore, we know that AWS must be using some kind of encapsulation / tunnelling so that VPC are possible. The Intel 82599 does not provide encapsulation support.

The solution then would be to do the extra processing elsewhere — either off the host or in the host, using a co-processor. This schematic shows processing happening at the TOR switch. The drawback is that even intra-host traffic has to be tromboned via the TOR. Furthermore the switch now becomes a pretty big bottleneck and a failure in the switch could lead to several hosts losing network connectivity.


Using a co-processor would be the best solution. Tilera is one such processor that comes to mind. Since the Tilera provides general purpose processing cores, the encap/decap/filtering/stateful firewall processing could be done in software instead of ASICs or FPGAs.


The software/hardware solution could allow AWS to introduce further innovations in its networking portfolio, including end-to-end encryption, IDS and IPS.

Disclaimer: I have no knowledge of AWS internals. This is just an exploration of “how did they build it?”.

Categories: FLOSS Project Planets

Edward J. Yoon: (기사 번역/의역) Hortonworks IPO: Where Are The Margins?

Thu, 2014-11-20 22:33
원문은 여기서: Forbes 뉴스 링크
호튼웍스의 SEC 상장 신청서 전문은 여기서: 링크

(내용 번역은 제 마음대로 요약/의역해서 작성합니다.)

호튼웍스: 어디서 남겨먹어?기업은 데이터로 움직이니, 단연코 요즘 가장 뜨거운 소프트웨어 전쟁터는 빅 데이터. 이 와중에, 호튼웍스가 미 증권거래위원회(SEC)에 상장 신청서를 제출하고 바짝 전투력을 끌어올리기 시작했다고.

그 전쟁터에는 메이저 3사가 있는데 그것이 바로 클라우데라, 맵알, 그리고 호튼웍스. 기 유사 시장 RDBMS 벤더들의 전쟁과는 다르게 이들은 오픈소스 하둡으로 데이터 플랫폼을 만들고 있는게 특징이라면 특징. 따라서, 전쟁의 승자는 누가 오픈소스 비지니스 모델을 잘 만드냐로 결판.

한편 상장 신청서에 따르면, 호튼웍스는 데이터 플랫폼을 무료로 제공하고 (역시 예상대로) 기술 지원 등으로 수익을 창출하겠다고.

오픈소스 사업에서는 기술 지원 등이 일반적인 모델이긴하나, 이는 기술 지원 인건비를 감안해야하므로 돈을 벌기 힘들다는게 중론 (한국형 SI같은 느낌). 게다가 HDP쓰는 사람들이 얼마나 돈을 쓸까에 대해 대단히 회의적.

특히나 경쟁사 클라우데라가 노골적인데 ㅋ 아무르 아와달라 클라우데라 최고기술책임자(CTO)는 호튼웍스는 지금 잘 못하고 있고 경쟁사를 방어할만한 무기가 없다며, 현재 돈을 벌기위해 얼마나 쓰고있는지 눈여겨 잘 보라고  은근슬적 디스아닌 디스같은 감상 평을 남김.

그렇다면, 클라우데라와 맵알은 그럼 얼마나 차별화된 수익 모델이 있는 것인가!? 에 대해서 이후 몇 블락 적어놓았는데 별반 크게 다를것 없어보이므로 과감히 생략!

Edward J. Yoon님 comment: 한국에서도 1세대 유사 사업 모델은 사실상 적자행진 및 SI사업으로 전락. 내가 알 수 없지만 클라우데라는 앱시장 개척을 나설 듯. 맵알은 아웃오브관심. 진정한 전쟁은 2년 후 다시 보게 될 거라고 호언장담. 이 댓글은 성지글이 된다!
Categories: FLOSS Project Planets

Timothy Chen: Kafka common addons

Thu, 2014-11-20 20:22

The blog post I wrote for Packt is now published on their blog, talks about some common tools people use with Kafka:


Categories: FLOSS Project Planets

Igor Galić: Open Sourcing Mental Illness - On Depression and Addiction

Thu, 2014-11-20 19:00

This talk runs under Ed Finkler‘s Banner of Open Sourcing Mental Illness, who has been running these types of talks for over a year now. Ed talks very broadly on the subject, while still backing it up with his personal stories.

I’d like to mirror his work under that banner: I’d like to talk about depression, and addiction: Open source is work. Not just a work of passion, but real actual work. And it can be taxing. I’d like to discuss how.

Content Warnings galore: The topics I discuss and the stories I tell can be triggering. I am perfectly happy answering questions, but not everyone will be comfortable asking. You can always seek me outside of this talk.


I’ll start with some introductions…

How I appear to be like everyone else (here)
  • english speaking
  • male (i had one woman in my talk)
  • white
  • straight
How I’m not quite like everyone else
  • war refugee
  • immigrant
  • foreigner
  • polyglot (Bosanski/српски/hrvatski, Deutsch, English, C, Ruby)
  • dyslexic
  • grew up in abusive home
  • school drop-out
  • alcoholic
  • autodidact
  • depression
  • workaholic
  • queer
  • probably a twitter addict

Why I’m telling you this

I could be telling you this for a number of reasons: first off, I could simply be lying. This could be a fabrication. And one way or another it is a manipulation: I could be trying to feed you subliminal messages about the importance of diversity, in order to further my feminist, queer agenda. But, actually, it’s about making myself vulnerable, about opening up. This is an extremely sensitive subject, so I will need you to trust me.

This trust comes with a caveat: I am not a health-care specialist. All I’m doing is sharing my own stories and experiences. What has worked for me, might not work for you. All I can do for you is listen to your stories, but the only real advice I’ll be able to give is to ask you to seek help.

Let’s get back on to my list: Many of those items can be “fixed”, other things are not for fixing but might need to be discovered and embraced.

I cannot go back and change the fact that as a child I had to flee my home country, and along with other things in my history, it made me the person who I am. I have “managed” my alcoholism in a simple, yet effective way: I quit.

This, however, is not so simple with other issues…


I am constantly reminded of the fact that depression is a chronic illness, one that can drastically shorten a person’s life-expectancy. These reminders come through the media and medical journals I read, and they come through personal experiences.

Last week, a friend of mine died

I have read, and re-read these words many times since the night I wrote them, and they remain painful.

There are no words from me that can fittingly describe the overpowering feeling of nothingness, of one’s own inadequacy, and the loneliness that makes up depression. Better artists have already done that.

All I can contribute is this

Depression lies, and when we listen to its lies, it kills.

I couldn’t just quit depression. Which, ridiculously enough, is something that people who don’t know depression will demand. The same kind of people who will happily ask me, every time, if I want a drink, while only listing alcohol.

There’s a lesson here, and I’ll come back to this subject later.

Step One: Self-treatment

Having been a life-long autodidact, I was very eager to make a changes to “fix my life”. I would listen to advice from friends or even random strangers. I’d take their advise on what bothered them about me, or I’d talk about something that was bothering me about myself.

The pattern was always the same, and it’s a well-documented in self-help books that are not complete garbage, because it’s extremely effective

  • Identify an issue
  • Find a set of solutions, and choose one
  • Implement it, and stick with it for >=3 months
  • Document, or observe changes
  • Verify the result

Every single of one those steps is extremely hard.

It’s well possible to train myself in a new habit of getting up an hour earlier, or working out three times a week to run a marathon, or to learn a new (programming) language. It’s extremely hard, but it’s possible.

It’s a lot harder however, if I were in a depressive episode, and my brain is lying to me.

  • Every issue will be with me.
  • Every solution will be that I somehow vanish from existence.
  • Every attempt at sticking with a plan will be abandoned within days.
  • Every observation will be warped.
  • Every reflexion will conclude that the issue was with me to begin with, and that, yes, I must somehow vanish from this plane of existence.

And, none of this will help me identify that I do indeed have depression. None of this did help me identify that. Sometimes you will need feedback from the out-side world. Some people get that through friends or family, who rightly recognize their issue, and have the courage to point it out. Maybe they even point it out in a helpful way.

This whole process is made all the harder in that depression is insulating me from the things that people are telling me, be that by overriding everything they say with:

They must be lying

Or, by directly isolating me from my friends.

My push past those lines of defense came through an eye-opening experience with a former partner.

Gauging your Limits

My then partner and I were together for about six months at that time, when she left for long-planned month long vacation to south-east Asia.

She came back a “different woman.” One she hadn’t been in over ten years: Anxiety and depression started to slowly seep in. They fed her insomnia. The lack of sleep made her work as primary school teacher even more stressful.

The simplest of tasks such as printing a word document became infuriatingly frustrating. Sending her into a fit of rage, or just sobbing and crying.

I was there for here; I’d work-out with her when she asked me to; I’d cook with her, help her with her paintings; I’d hold her. But none of the help I thought I could provide was enough.

She started researching suicide methods. There was nothing I could do.

All I could do was ask her to seek help. And help her get bootstrapped: Find therapy. Find a psychiatrist. Make appointments.

I hit the cold hard wall of reality, and it made me realize: That was the limit of my ability. It also made me realize that she was going through experiences that were anything but new to me. She was experiencing it on a magnitude that I never had. But that only helped to reinforce my own powerlessness.

There was nothing I could do, yet there was a support network ready to help her.

If there was help for her, then surely there was help for me too.

Being Out In the Open

Three years ago, I had my “coming out”. I’m all about Open Source. I know the benefits of doing things in the open.

With enough eyes, all bugs become shallow

If there’s any message you take away from this talk, let it be that:

No, This is not true.[1]

Heck, it’s not even true for Open Source, as OpenSSL has so thoroughly demonstrated.

Even if my friends were actually medical health professionals, I couldn’t possibly expect help from them directly.

But there are other benefits from being out openly:

It’s much easier to create a support-network

The most important aspect of being out openly to my friends, means that my friends already know. This means that when I need their help, or need them to be more sensitive, or need more room, then I don’t need to have that conversation. We’ve already had it.

When I fall behind on my goals…

…people can be there for me. They can remind me of those goals. Or they might offer support, such as finding a new therapist.

I’d like to quote Avery Edison on the subject of coming out, from her book, Right Body, Wrong Junk[2]. Avery is a queer woman, and her book was on queer issues, but the following commentary on her own writing still struck a chord with me:

The fact that it’s not a secret makes it all the easier to say. I still worry about how someone might react (there’s always the possibility of a hidden bigot in the mix), but the whole process is easier. I don’t feel like I’m unburdening myself, I just feel like I’m providing relevant info.

But, unlike coming out, which in my experience is not that big a deal anymore, there’s this giant stigma attached to mental illness, like all things that only exist in the margins of society, so disclosing it is a delicate issue.


Let’s talk about addiction then.


In January 2012 I started working as freelancer, and for the first time I was earning adequate amounts of money for my level of experience and the kind of performance I was delivering. Which was really good, because after six months of unemployment, of traveling, and, drinking, I had amassed around €20000 in debts.

Those first two months, in which I worked my ass off, and, automated myself away, had paid off my debt. But, I still hadn’t quite learned any lesson at that point. It took another six months of drinking, and “writing” (sitting in bars with a notebook), until a police patrol picked me off my bike after a couple Guinness too many. Once again, this was an external incentive, one for which, despite the big price tag, I will be eternally grateful for.

It gave me official reason to pause, and reflect. Reflect on things I had suspected were wrong for a long time. However, I had pushed them aside, as being under my control, or important for my creativity, or that it’s genetic, or part of my charming personality.

I decided to quit. To literally never drink again, and that on its own was an admission: I had admitted to myself that I was an addict. But that was just a shaky start: Two weeks down the road, OSCon was going to be on, and I was deeply afraid of what I knew was going to be obviously happening: An Open Source Event meant it was guaranteed that beer and booze would be flowing in near infinite amounts.

Open Booze

Let’s take a minute to talk about our tiny fragment of our culture here. Fortunately I don’t have to pull anything out of my ass, because the outstanding Kara Sowles wrote a piece for Model View Culture’s Events issue about Alcohol and Inclusivity: Planning Tech Events with Non-Alcoholic Options.

To very, very, very briefly summarize what everyone should read:

  • There are people who don’t drink alcohol
  • They don’t have to give you a reason
  • Events should provide them with drinking options

I am deeply indebted, to Nóirín Plunkett for standing up for me, and defending my decision before other OSCon participants, who were about to ask the dreaded


Nóirín’s ferocity gave me the courage to stand up for myself and simply say:

Because I’m an alcoholic.

When I start, I don’t stop. So I decided not to start.

In just a couple of days after quitting, I had learned how powerful the truth about addiction can be. When it’s simply pronounced like that, it left people speechless. There is some dark irony behind the fact that I used to drink, because I thought I am unable to talk to people otherwise[3]. I was also afraid, that my not drinking would be the only subject to talk about. Fortunately, that turned out not to be the case. And, at the last conference party I was at (organized by Kara, no less!) I had someone call me a

Social Butterfly

But if you read Kara’s article you may learn that this kind of openness isn’t for everyone. It can be hard enough to admit this to ourselves, or admitting it to friends and family, but imagine how hard it is then to admit it to all kinds of strangers, over, and over, and over, and over again.

This is again a parallel to coming out as queer: Unless we have it tattooed across our foreheads, queer people don’t have their “coming out” once, we have to come out over, and over, and over again to new friends, new colleagues, or potential love interests. As you can guess, some of those coming outs are easier than others…

Let’s talk about work

For the longest part of my life, my work has been a core part of my identity.[4]

Šta radiš, Igo?


If I hadn’t been really, really bad at sucking up to authority, I might have made it all the way through school and into university, and might now be an ACADEMIC.

maybe i am, without knowing it, an academic(!!!).

— The Wrath of PB™ (@hirojin) October 13, 2014

But even so, I am doing something (for money), that I absolutely enjoy.

i should start using twitter for time keeping:
✶ complaining about meetings → meeting
✶ complaining about rspec / puppet → "open source"
✶ complaining about haskell / python / java / puppet / configuration formats → ~paid work~
✶ complaining about debuntu / ipv6 → @brainsware

— The Wrath of PB™ (@hirojin) October 28, 2014

Except, when I don’t. Make money for it, I mean.

A Show of Hands
  • Who here does no Open Source Work at all?
  • Who here lives 100% off their Open Source work?
  • 50ish%?
  • Who here would starve if all they did was Open Source?

[Update: Very few people dared to raise their hands.]

For a number of personal reasons, I fall into that last category. We’ll come back to this eventually, but first I’d like to about my

First Burn Out

When I was 21, I had my first, full-blown burn out. Seven years later, I would write

The first time I had a burn out while still at school, at the age of 21. It was essentially three months of me not leaving the bed.

Of course, that’s not true. Almost a decade later, I would ask my mother about the time-line, and it turns out it was more like a week or so. While I’m bad with time, not even I am that bad with time.

Time is an illusion. Lunch-time doubly so.

In its final stages, burnout can cause a full-blown depression. But the first step down-hill is

Working Harder

Because they have to prove themselves to others or try to fit in an organization that does not suit them, people establish high personal expectations.

I spent seven years in a five-year school, failing every odd grade. And yet, even after having had a complete burn-out, I wanted to come back and finish it. Only then, and, thanks to being employed and having slipped into a chronic lack of time, I realized that school wasn’t quite for me. Finally.

Second Burn Out

I spent two years too long in a job I couldn’t stand. The company had grown too big, the management too deep. As Engineer, I was earning €30 more than when I started as Junior. Except when I was on-call.

I stayed long hours at the office, staring blankly at the screen, or “doing open source” (IRC, before there was GitHub and Twitter), until everyone had left, so I could finally get some work done, when it was silent.

Often, even then, I couldn’t really get anything done, because of course I depended on my colleagues to do things for me. So all I’d do would be prepare my work, and write passive aggressive emails or tickets. Then, cycle home super late, sleep in long, lie half-asleep for two hours in bed, and get in super late to work.

This turned out to be an unsustainable cycle[5]: I quit my job.

Now, quitting your job is not always an option. Quitting work in general, appears to be no option at all in our society. So, after a six month hiatus, I finally became self-employed. And that changed things considerably for me:

First, I could set my own price. But more importantly, I am now only getting paid for things I’m actually doing. There’s no need to be present 40 hours a week. When I’m unproductively staring at the screen, I’m not billing my clients, and that is strangely liberating.[6]

And, as I have recently discovered that, when projects are piling up, developers ~getting sick~, management being conspicuously vague or absent, I can simply remove myself for a week.

I used to be proud of being able to work in a stressful environment. Now I’m just really happy when I can work in one that’s not toxic.

Which leads us right into the topic of Open Source.

A Letter from a Hobbyist

I could probably count the jobs and projects I’ve gotten through my Open Source work on one hand. I’m not a developer in the classical sense. And on the DevOps continuum, I am definitely on the Ops side. For a long time I’ve seen Open Source as a hobby. An intellectually slightly more challenging and thus rewarding pursuit, than, say, reading comics.

However, Open Source is Work.

If Open Source is work, and if, for many people it’s unpaid work, then it’s hard for it to be a hobby that provides an adequate counter-weight to your job. It becomes another stressor. And we soon find ourselves trying to avoid it…

The problem isn’t really the software we’re working on…

the short summary is that software is suffering.
the longer story is that software is suffering, because people.

— The Wrath of PB™ (@hirojin) November 13, 2014

J. Random Hacker

A quick show of hands, who here has read esr’s

[Update: surprisingly few people knew this, but many remembered after my talk]

If you haven’t read the latter, I highly recommend to also compare it with the WikiPedia on the burnout “roadmap”[7]

Probably the biggest disconnect with the rest of the text is visible from Weaknesses chapter.

Hackers have relatively little ability to identify emotionally with other people. This may be because hackers generally aren’t much like ‘other people’. Unsurprisingly, hackers also tend towards self-absorption, intellectual arrogance, and impatience with people and tasks perceived to be wasting their time.

I would posit that this portrait of J. Random Hacker was probably the most toxic piece of writing that I could have absorbed while growing up and learning to program. It validated all the things that lead me into burning up, and it brushed aside all the things that I was uncomfortable with.

Making a Change

I believe that our ~hacker culture~ is in urgent need of disruption. I am convinced that it’s so thoroughly toxic that we are going to work ourselves, and the people who most depend on us, into a corner. For now, while we are still the movers and the shakers of the world, we should perhaps stop and consider how to wield that power responsibly.

If you do not suffer from depression or addiction

I think the first and simplest thing that everyone can do is to be more considerate of themselves, and of others.

Leslie Hawthorn has held this very short keynote at OSCon this year, in which she gives a very simple recipe for how to do that:

Listen to people who are not like you.

If you suffer from depression or addiction

I can only recommend you surround yourself with people who support you, not just with people who claim they are supportive of you.

And, I implore you, seek help.

I am convinced that if everyone in this room today were to regularly see a therapist, that the world would be a better place tomorrow.

Thank you.

Resources Thanks

[1]: writing those lines was extremely hard. Just writing it, despite its context, felt absolutely hypocritical, after recent experiences with a friend dying.

[2]: I’m very thankful to @kangarara for pointing out me that co-locating queer issues and mental health might make people associate that queerness is a mental illness. If my words on the subject can’t help clarify that this is not what I mean, it’s probably better to delete this whole section in future revisions.

[3]: do I really have to point out how badly that worked?

[4]: There is another 4000 word essay in that sentence, that I didn’t get to write last week, for reasons already mentioned…

[5]: “unsustainable cycle is a euphemism: I was suicidal at that time.

[6]: There’s this piece by Quinn Norton, Against Productivity, that I still haven’t read…

[7]: thanks to David Schmitt for pointing out that connection

[X]: TXT: more txt

Categories: FLOSS Project Planets

Justin Mason: Links for 2014-11-20

Thu, 2014-11-20 18:58
  • Update on Azure Storage Service Interruption

    As part of a performance update to Azure Storage, an issue was discovered that resulted in reduced capacity across services utilizing Azure Storage, including Virtual Machines, Visual Studio Online, Websites, Search and other Microsoft services. Prior to applying the performance update, it had been tested over several weeks in a subset of our customer-facing storage service for Azure Tables. We typically call this “flighting,” as we work to identify issues before we broadly deploy any updates. The flighting test demonstrated a notable performance improvement and we proceeded to deploy the update across the storage service. During the rollout we discovered an issue that resulted in storage blob front ends going into an infinite loop, which had gone undetected during flighting. The net result was an inability for the front ends to take on further traffic, which in turn caused other services built on top to experience issues. I’m really surprised MS deployment procedures allow a change to be rolled out globally across multiple regions on a single day. I suspect they soon won’t.

    (tags: change-management cm microsoft outages postmortems azure deployment multi-region flighting azure-storage)

  • AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems – YouTube

    This is a really solid talk — not surprising, alv@ is one of the speakers!

    “AWS and operate some of the world’s largest distributed systems infrastructure and applications. In our past 18 years of operating this infrastructure, we have come to realize that building such large distributed systems to meet the durability, reliability, scalability, and performance needs of AWS requires us to build our services using a few common distributed systems primitives. Examples of these primitives include a reliable method to build consensus in a distributed system, reliable and scalable key-value store, infrastructure for a transactional logging system, scalable database query layers using both NoSQL and SQL APIs, and a system for scalable and elastic compute infrastructure. In this session, we discuss some of the solutions that we employ in building these primitives and our lessons in operating these systems. We also cover the history of some of these primitives — DHTs, transactional logging, materialized views and various other deep distributed systems concepts; how their design evolved over time; and how we continue to scale them to AWS. “ Slides:

    (tags: scale scaling aws amazon dht logging data-structures distcomp via:marc-brooker dynamodb s3)

  • How Curiosity, Luck, and the Flip of a Switch Saved the Moon Program | Motherboard

    “SCE to off?” someone said. The switch was so obscure that neither of his bosses knew what he was talking about. “What the hell’s that,” blurted out Gerald Carr, who was in charge of communicating with the capsule. The rookie flight director, Gerry Griffin, didn’t know either. Sixty seconds had passed since the initial lightning strike. No one else knew what to do. The call to abort was fast approaching.  Finally, Carr reluctantly gave the order in a voice far cooler than the moment. “Apollo 12, Houston, try SCE to Auxiliary, over.”

    (tags: spaceflight stories apollo sce-to-aux power lightning weather outages simulation training nasa)

Categories: FLOSS Project Planets

Bryan Pendleton: Drip drip drip

Thu, 2014-11-20 18:14

It's snowing in Truckee!.

And it rained about half an inch, maybe more, here in the Bay Area, today.

But oh-my-goodness is it dry: 203 drought maps reveal just how thirsty California has become.

More than 80% of California is in extreme drought, according to the U.S. Drought Monitor, and the state's condition isn't expected to improve in the near future.

I'm not sure I'd want to change places with Buffalo, New York, though: Jets-Bills Game Will Not Be Played In Buffalo Due To Massive Snowfall.

A league spokesman says a severe lake-effect snowstorm hitting Buffalo has led the NFL to relocate the Bills home game against the New York Jets on Sunday.

But hopefully we can at least continue to get some rain...

Categories: FLOSS Project Planets

Bryan Pendleton: Things you can put into your roasted vegetables dish

Wed, 2014-11-19 21:13

For example, if you're making roasted vegetables for the company potluck.

  • Sweet Potatoes
  • Yams
  • Potatoes
  • Parsnips
  • Beets
  • Carrots
  • Celery
  • Leeks
  • Butternut Squash
  • Acorn Squash
  • Kabocha Squash

Clean; peel; chop.

Toss with olive oil, thyme, rosemary, salt and pepper.

Bake at 350 degrees for 45 minutes or so.

Attend company potluck.

Categories: FLOSS Project Planets

Justin Mason: Links for 2014-11-19

Wed, 2014-11-19 18:58
  • Building a complete Tweet index

    Twitter’s new massive-scale twitter search backend. Sharding galore

    (tags: architecture search twitter sharding earlybird)

  • The Oral History Of The Poop Emoji (Or, How Google Brought Poop To America)

    ‘I went over to Japan right around the time Takeshi was deciding which emoji were going to make it into the first cut of Gmail emoji. The [PILE_OF_POO emoji] was absolutely one of the necessary emoji that Takeshi said we have to have. There was actually conflict because there were people back at headquarters who had no idea what emoji were, and thought that having an animated [turd] in their Gmail was offensive.’ ‘[The poop emoji] got very popular when a comic called “Dr. Slump” was broadcast in Japan back to the ‘90s. Such poop was not an object to be disliked, but it had a funny meaning. This was a very popular comedy animation where a girl played a trick on other people using the poop. The poop was this funny object to play with. It was never serious.’ ‘In Japanese that’s called “unchi.” It’s a child word with a benign meaning. ‘

    (tags: culture emoji google pile-of-poo turd poo japan gmail unchi dr-slump)

  • LUNAR MISSION ONE: A new lunar mission for everyone. by Lunar Missions Ltd — Kickstarter

    We plan to send an unmanned robotic landing module to the South Pole of the Moon – an area unexplored by previous missions. We’re going to use pioneering technology to drill down to a depth of at least 20m – 10 times deeper than has ever been drilled before – and potentially as deep as 100m. By doing this, we will access lunar rock dating back up to 4.5 billion years to discover the geological composition of the Moon, the ancient relationship it shares with our planet and the effects of asteroid bombardment. Ultimately, the project will improve scientific understanding of the early solar system, the formation of our planet and the Moon, and the conditions that initiated life on Earth. Kickstarter-funded — UKP 600k goal. Just in time for xmas!

    (tags: kickstarter science moon lunar-mission-one exploration)

  • Flow, a new static type checker for JavaScript

    Unlike the (excellent) Typescript, it’ll infer types:

    Flow’s type checking is opt-in — you do not need to type check all your code at once. However, underlying the design of Flow is the assumption that most JavaScript code is implicitly statically typed; even though types may not appear anywhere in the code, they are in the developer’s mind as a way to reason about the correctness of the code. Flow infers those types automatically wherever possible, which means that it can find type errors without needing any changes to the code at all. On the other hand, some JavaScript code, especially frameworks, make heavy use of reflection that is often hard to reason about statically. For such inherently dynamic code, type checking would be too imprecise, so Flow provides a simple way to explicitly trust such code and move on. This design is validated by our huge JavaScript codebase at Facebook: Most of our code falls in the implicitly statically typed category, where developers can check their code for type errors without having to explicitly annotate that code with types.

    (tags: facebook flow javascript coding types type-inference ocaml typescript)

  • Exactly-Once Delivery May Not Be What You Want

    An extremely good explanation from Marc Brooker that exactly-once delivery in a distributed system is very hard.

    And so on. There’s always a place to slot in one more turtle. The bad news is that I’m not aware of a nice solution to the general problem for all side effects, and I suspect that no such solution exists. On the bright side, there are some very nice solutions that work really well in practice. The simplest is idempotence. This is a very simple idea: we make the tasks have the same effect no matter how many times they are executed.

    (tags: architecture messaging queues exactly-once-delivery reliability fault-tolerance distcomp marc-brooker)

  • GpsMapIreland

    This topographic map represents Ireland. It is designed for “hillwalking”. The contour lines are extracted from SRTM public data provided by NASA. These files contain a digitized ground represented by points. The sample rate defines a grid resolution for Ireland around 90m in northing and 60m in easting. In major cases, digitized points do not correspond with summits. Carrauntoohil (1039m, the highest summit of Ireland) does not appear in SRTM data. The altitude reaches only 1018m. Data were obtain from space with a radar. Because of the relative position between the radar and the earth, a shadow appears in some conditions (along ridges, behind summits…). This shadow matches with a gap in data (Imagine you with a flashlight in a dark room. It is hard to see what is in shadow). To close these gaps, you need other data or you can do interpolation. The second solution is applied in our case. There is one square degree per SRTM file with a sample rate of 1200×1200 points/square degree at Ireland latitude. [...] All in all you obtain contour lines pretty sufficient for walking.

    (tags: hillwalking walking ireland gps garmin open-data srtm maps hiking via:alan)

  • castnow

    Marcus Ramberg says: ‘If you have a chromecast and you’re not using castnow, I don’t know what is wrong with you.’

    (tags: chromecast castnow media movies tv video)

  • The Infinite Hows, instead of the Five Whys

    John Allspaw with an interesting assertion that we need to ask “how”, not “why” in five-whys postmortems:

    “Why?” is the wrong question. In order to learn (which should be the goal of any retrospective or post-hoc investigation) you want multiple and diverse perspectives. You get these by asking people for their own narratives. Effectively, you’re asking “how?“ Asking “why?” too easily gets you to an answer to the question “who?” (which in almost every case is irrelevant) or “takes you to the ‘mysterious’ incentives and motivations people bring into the workplace.” Asking “how?” gets you to describe (at least some) of the conditions that allowed an event to take place, and provides rich operational data.

    (tags: ops five-whys john-allspaw questions postmortems analysis root-causes)

  • the JVM now supports globbing in classpath specifications

    hooray, no more uberjars or monster commandlines!

    (tags: java jvm globbing classpath uberjars jars deployment)

  • Microsoft Azure 9-hour outage

    ‘From 19 Nov, 2014 00:52 to 05:50 UTC a subset of customers using Storage, Virtual Machines, SQL Geo-Restore, SQL Import/export, Websites, Azure Search, Azure Cache, Management Portal, Service Bus, Event Hubs, Visual Studio, Machine Learning, HDInsights, Automation, Virtual Network, Stream Analytics, Active Directory, StorSimple and Azure Backup Services in West US and West Europe experienced connectivity issues. This incident has now been mitigated.’ There was knock-on impact until 11:00 UTC (storage in N Europe), 11:45 UTC (websites, West Europe), and 09:15 UTC (storage, West Europe), from the looks of things. Should be an interesting postmortem.

    (tags: outages azure microsoft ops)

Categories: FLOSS Project Planets

Chris Hostetter: Stump The Chump: D.C. Winners

Wed, 2014-11-19 17:55

Last week was another great Stump the Chump session at Lucene/Solr Revolution in DC. Today, I’m happy to anounce the winners:

  • First Prize: Jeff Wartes ($100 Amazon gift certificate)
  • Second Prize: Fudong Li ($50 Amazon gift certificate)
  • Third Prize: Venkata Marrapu ($25 Amazon gift certificate)

Keep an eye on the Lucidworks YouTube page to watch the video as soon as it is available and see the winning questions.

I want to thank everyone who participated — either by sending in your questions, or by being there in person to heckle me. But I would especially like to thank the judges, and our moderator Cassandra Targett, who had to do all the hard work preparing the questions.

See you next year!

The post Stump The Chump: D.C. Winners appeared first on Lucidworks.

Categories: FLOSS Project Planets

Bryan Pendleton: Read, read, read

Tue, 2014-11-18 23:23

My youngest child is 23 years old!

Does this mean I am finally a grown-up?

  • Alexander Grothendieck 1928–2014So what can all this mean for us who work in what Grothendieck described as a “mansion” in which “the windows and blinds are all closed,” while he was one of those “whose spontaneous and joyful vocation it has been to be ceaseless building new mansions”? At least he did not call our dwelling a cave. However, in complexity theory we have it worse than Plato’s cave-prisoners in not merely missing the blinding world outside, but sensing its impact as a negative image in our present ignorance of lower bounds.
  • Andrew Odlyzko Strikes AgainLast year I blogged about Andrew Odlyzko's perceptive analysis of the business of scholarly publishing. Now he's back with an invaluable, must-read analysis of the economics of the communication industry entitled Will smart pricing finally take off?.
  • A Worm's Mind In A Lego BodyThe nematode worm Caenorhabditis elegans (C. elegans) is tiny and only has 302 neurons. These have been completely mapped and the OpenWorm project is working to build a complete simulation of the worm in software. One of the founders of the OpenWorm project, Timothy Busbice, has taken the connectome and implemented an object oriented neuron program.
  • Keep Daly City UncoolAdmittedly, the rumor that a proposed move to Daly City caused dissent among Reddit's ranks is met with a sense of relief around here where there's reason to fear that all those nearby gobs of money will ooze out of San Francisco and Atherton and engulf Val's, Joe's and our twin Targets like ravenous blob fueled by Bitcoin and fully-vested stock options.
  • Introducing “A Field Guide to the Distributed Development Stack”: Tools to develop massively distributed applicationsIn addition to being a (hopefully) useful framework, the Guide is also meant to be a living resource. So, we’ve put the source on GitHub and invite you to contribute. If you feel like we’ve missed a tool (which we most certainly have, since new things are popping up every day) or a major theme, then fork the repo and send me a pull request. We’ll be keeping this document up to date and republishing it as we watch this trend continue to grow. We’ll use O’Reilly Atlas to pull in the contributions and periodically republish the guide.
  • Binary artifact management in GitManagement of large binaries is still an unsolved problem in the Git community. There are effective alternatives and work-arounds but it’ll be interesting to see if anyone tries to solve the problem more systematically.
  • Beating Ebola Means Drinking, Last Thing Patient Wants to Do“We will halve the mortality by firstly just stopping anti-inflammatories and giving hydration, and really pushing it,” Mardel said. “I want every man and woman in Sierra Leone to know this. I want sports personalities to be talking about it. I want everybody to be talking about it.”
  • This Mod Is the Absolute Best Way to Play XCOMXCOM: Enemy Unknown is a great tactical game from Firaxis games. Its first expansion pack, Enemy Within, added so much amazing content that it made playing the game an entirely new experience. And now there’s a mod that’s so good it makes the expansion pack feel like a tired retread.
Categories: FLOSS Project Planets

Bryan Pendleton: This time, the buzzing was not in my head

Tue, 2014-11-18 20:49

Reason number forty nine to have an unnecessary panic attack:

The other evening, I was going through my house, getting ready for bed, turning off lights, closing doors, making all my regular routine rounds.

I check the garage to make sure the light is off, but when I open the door, there is a strange sound.

It's sort of a buzzing sound, sort of a clicking sound, sort of a humming sound.

I wander around the garage for a while, trying to find the source of the sound: it's clearly coming from the car.

The car is switched off, but I wonder: did I perhaps leave the key in the ignition? leave a switch at a funny setting?


The sound is coming from under the back of the car, so I get down on my tummy, shine my flashlight around, stick my arm in and try to touch various parts of the car to see if any of them are vibrating.

It's completely baffling.

So, I do the Only Sensible Thing: I go back in to the house, sit down at my computer, and search the Web.

Sure enough, there are dozens of forums filled with discussions of this behavior, and after clicking a few times, one of the messages sends me back out to the car, to look in the glove compartment, in the Owner's Manual, where, on page 9, I find:

Noise from under the vehicle.

NOTE: You may hear a noise from under the vehicle approximately 5 to 10 hours after the engine is turned off. However, this does not indicate a malfunction. This noise is caused by the operation of the fuel evaporation leakage checking system and is normal. The noise will stop after approximately 15 minutes.

And it did.

Categories: FLOSS Project Planets

Justin Mason: Links for 2014-11-18

Tue, 2014-11-18 18:58
Categories: FLOSS Project Planets

Justin Mason: Links for 2014-11-17

Mon, 2014-11-17 18:58
  • FBI’s “Suicide Letter” to Dr. Martin Luther King, Jr., and the Dangers of Unchecked Surveillance

    The entire letter could have been taken from a page of GCHQ’s Joint Threat Research and Intelligence Group (JTRIG)—though perhaps as an email or series of tweets. The British spying agency GCHQ is one of the NSA’s closest partners. The mission of JTRIG, a unit within GCHQ, is to “destroy, deny, degrade [and] disrupt enemies by discrediting them.” And there’s little reason to believe the NSA and FBI aren’t using such tactics. The implications of these types of strategies in the digital age are chilling. Imagine Facebook chats, porn viewing history, emails, and more made public to discredit a leader who threatens the status quo, or used to blackmail a reluctant target into becoming an FBI informant. These are not far-fetched ideas. They are the reality of what happens when the surveillance state is allowed to grow out of control, and the full King letter, as well as current intelligence community practices illustrate that reality richly.

    (tags: fbi surveillance mlk history blackmail snooping gchq nsa)

  • WriterReaderPhaser

    A nice new concurrency primitive from Gil Tene:

    Have you ever had a need for logging or analyzing data that is actively being updated? Have you ever wanted to do that without stalling the writers (recorders) in any way? If so, then WriterReaderPhaser is for you.  I’m not talking about logging messages or text lines here.  I’m talking about data.  Data larger than one word of memory.  Data that holds actual interesting state. Data that keeps being updated, but needs to be viewed in a stable and coherent way for analysis or logging.  Data like frame buffers. Data like histograms.  Data like usage counts. Data that changes. see also Left-Right:

    (tags: phasers data-structures concurrency primitives algorithms performance wait-free)

  • 3D Secure and Verified By Visa to be canned


    Mastercard and Visa are removing the need for users to enter their passwords for identity confirmation as part of a revamp of the existing (oft-criticised) 3-D Secure scheme. The arrival of 3D Secure 2.0 next year will see the credit card giants moving away from the existing system of secondary static passwords to authorise online purchases, as applied by Verified by Visa and MasterCard SecureCode, towards a next-gen system based on more secure biometric and token-based prompts. (via Gordon)

    (tags: via:gsyme verified-by-visa 3d-secure mastercard visa credit-cards authentication authorization win passwords)

  • Aeron: Do we really need another messaging system? – High Scalability

    excellent writeup on Aeron

    (tags: aeron messing libraries java martin-thompson performance mechanical-sympathy queueing ipc tcp)

Categories: FLOSS Project Planets

Nick Kew: A great venue

Mon, 2014-11-17 07:39

It’s lunchtime on the first day of Apachecon.  Too soon to assess the event as a whole, but I’ve formed a view on the venue.

Of all the ApacheCon venues I’ve been to, I think this week’s seems the best.  The Corinthia Hotel is about as good as any I’ve encountered, and we’re in a nice area of the great historic city of Budapest.  Amsterdam is the only past-Apachecon city that can really rival Budapest, but that was let down by a bad conference hotel.  And conversely, where I’ve encountered decent hotels, they’ve been in some altogether less pleasant or interesting locations.  At worst we’ve had poor hotels in poor locations.

Come to think of it, that’s not just Apachecon, it’s conferences of any kind, even stretching back to my days in academia.

Of course, my perception may be coloured by individual circumstances too.  I’m not doing anything stressful like giving a talk or tutorial this time.  And I may have been fortunate to have been allocated an ideal hotel room, overlooking a quiet quadrangle where I can open the window wide for fresh air without being disturbed either by outside traffic or hotel noise.

Just a couple of flies in the ointment.  The weather in bleak November isn’t entirely conducive to getting the most from Budapest.  And there are not sufficient power outlets to wield the laptop everywhere around the conference.  Even if that’s (arguably) a good thing when in a presentation, the shortage of power points applies even to the designated hacker area, which is itself not a strong point of the event.

OK, time to get back to conferring!

Categories: FLOSS Project Planets

Nick Kew: A great venue

Mon, 2014-11-17 07:39

It’s lunchtime on the first day of Apachecon.  Too soon to assess the event as a whole, but I’ve formed a view on the venue.

Of all the ApacheCon venues I’ve been to, I think this week’s seems the best.  The Corinthia Hotel is about as good as any I’ve encountered, and we’re in a nice area of the great historic city of Budapest.  Amsterdam is the only past-Apachecon city that can really rival Budapest, but that was let down by a bad conference hotel.  And conversely, where I’ve encountered decent hotels, they’ve been in some altogether less pleasant or interesting locations.  At worst we’ve had poor hotels in poor locations.

Come to think of it, that’s not just Apachecon, it’s conferences of any kind, even stretching back to my days in academia.

Of course, my perception may be coloured by individual circumstances too.  I’m not doing anything stressful like giving a talk or tutorial this time.  And I may have been fortunate to have been allocated an ideal hotel room, overlooking a quiet quadrangle where I can open the window wide for fresh air without being disturbed either by outside traffic or hotel noise.

Just a couple of flies in the ointment.  The weather in bleak November isn’t entirely conducive to getting the most from Budapest.  And there are not sufficient power outlets to wield the laptop everywhere around the conference.  Even if that’s (arguably) a good thing when in a presentation, the shortage of power points applies even to the designated hacker area, which is itself not a strong point of the event.

OK, time to get back to conferring!

Categories: FLOSS Project Planets

Michael McCandless: Apache Lucene™ 5.0.0 is coming!

Mon, 2014-11-17 06:11
At long last, after a strong series of 4.x feature releases, most recently 4.10.2, we are finally working towards another major Apache Lucene release!

There are no promises for the exact timing (it's done when it's done!), but we already have a volunteer release manager (thank you Anshum!).

A major release in Lucene means all deprecated APIs (as of 4.10.x) are dropped, support for 3.x indices is removed while the numerous 4.x index formats are still supported for index backwards compatibility, and the 4.10.x branch becomes our bug-fix only release series (no new features, no API changes).

5.0.0 already contains a number of exciting changes, which I describe below, and they are still rolling in with ongoing active development.

Stronger index safety

Many of the 5.0.0 changes are focused on providing stronger protection against index corruption.

All file access now usesJava's NIO.2 APIs, giving us better error handling (e.g., Files.delete returns a meaningful exception) along with atomic rename for safer commits, reducing the risk of hideous "your entire index is gone" bugs like this doozie.

Lucene's replication module, along with distributed servers on top of Lucene such as Elasticsearchor Solr, must copy index files from one place to another. They do this for backup purposes (e.g., snapshot and restore), for migrating or recovering a shard from one node to another or when adding a new replica. Such replicators try to be incremental, so that if the same file name is present, with the same length and checksum, it will not be copied again.

Unfortunately, these layers sometimes have subtle bugs (they are complex!). Thanks to checksums(added in 4.8.0), Lucene already detects if the replicator caused any bit-flips while copying, and this revealed a long standing nasty bug in the compression library Elasticsearch uses.

With 5.0.0 we take this even further and now detect if whole files were copied to the wrong file name, by assigning a unique id to every segment and commit (segments_N file). Each index file now records the segment id in its header, and then these ids are cross-checked when the index is opened.

The new Lucene50Codecalso includes further index corruption detection.

Even CorruptIndexException itself is improved! It will now always refer to the file or resource where the corruption was detected, as this is now a required argument to its constructors. When corruption is detected higher up (e.g., a bad field number in the field infos file), the resulting CorruptIndexExceptionwill now state whether there was also a checksum mismatch in the file, helping to narrow the possible source of the corruption.

Finally, during merge, IndexWriter now always checks the incoming segments for corruption before merging. This can mean, on upgrading to 5.0.0, that merging may uncover long-standing latent corruption in an older 4.x index.

Reduced heap usage

5.0.0 also includes several changes to reduce heap usage during indexing and searching.

If your index has 1B docs, then caching a single FixedBitSet-based filter in 4.10.2costs a non-trivial 125 MB of heap! But with 5.0.0, Lucene now supports random-writable and advance-able sparse bitsets (RoaringDocIdSetand SparseFixedBitSet), so the heap required is in proportion to how many bits are set, not how many total documents exist in the index. These bitsets also greatly simplify how MultiTermQuery is rewritten (no more CONSTANT_SCORE_AUTO_REWRITE_METHOD), and they provide faster advance implementations than FixedBitSet's linear scan. Finally, they provide a more accurate cost() implementation, allowing Lucene to make better choices about how to drive the intersection at query time.

Heap usage during IndexWriter merging is also much lower with the new Lucene50Codec, since doc values and norms for the segments being merged are no longer fully loaded into heap for all fields; now they are loaded for the one field currently being merged, and then dropped.

The default norms format now uses sparse encoding when appropriate, so indices that enable norms for many sparse fields will see a large reduction in required heap at search time.

An explain API for heap usage

If you still find Lucene using more heap than you expected, 5.0.0 has a new API to print a tree structure showing a recursive breakdown of which parts are using how much heap. This is analogous to Lucene's explain API, used to understand why a document has a certain relevance score, but applied to heap usage instead.

It produces output like this:

_cz(5.0.0):C8330469: 28MB
postings [...]: 5.2MB
field 'latitude' [...]: 678.5KB
term index [FST(nodes=6679, ...)]: 678.3KB
This is a much faster way to see what is using up your heap than trying to stare at a Java heap dump.

Further changes

There is a long tail of additional 5.0.0 changes; here are some of them:

  • Old experimental postings formats (Sep/Fixed/VariableIntPostingsFormat) have been removed. PulsingPostingsFormat has also been removed, since the default postings format already pulses unique terms.

  • FieldCache is gone (moved to a dedicated UninvertingReader in the misc module). This means when you intend to sort on a field, you should index that field using doc values, which is much faster and less heap consuming than FieldCache.

  • Tokenizers and Analyzers no longer require Reader on init.

  • NormsFormat now gets its own dedicated NormsConsumer/Producer

  • Simplifications to FieldInfo (Lucene's "low schema"): no more normType (it is always a DocValuesType.NUMERIC), no more isIndexed (just check IndexOptions)

  • Compound file handling is simpler, and is now under codec control.

  • SortedSetSortField, used to sort on a multi-valued field, is promoted from sandbox to Lucene's core

  • PostingsFormat now uses a "pull" API when writing postings, just like doc values. This is powerful because you can do things in your postings format that require making more than one pass through the postings such as iterating over all postings for each term to decide which compression format it should use.

  • Version is no longer required on init to classes like IndexWriterConfig and analysis components.

The changes I've described here are just a snapshot of what we have lined up today for a 5.0.0 release. 5.0.0 is still under active development (patches welcome!) so this list will change by the time the actual release is done.
Categories: FLOSS Project Planets

Bryan Pendleton: Carlsen wins game 6!

Sun, 2014-11-16 11:46

After 6 rounds, it's Magnus Carlsen 3.5, Viswanathan Anand 2.5.

Superb chess, even though it's nearly all over my head. You can read lots of analysis on the web, including here.

Until I read the analysis, I didn't understand exactly where Anand lost, but I'm comforted that after I read the analysis, I understood it.

Me, I just love the fact that they both play 1. e4.

Categories: FLOSS Project Planets

Matt Raible: The House

Sun, 2014-11-16 11:12

A few months after starting this blog, I wrote about The Cabin. I grew up in a the cabin in the backwoods of Montana, with no electricity and no running water. I lived there for 16 years before moving to Oregon for my last two years of high school. As you can imagine, this makes for a good story now that I'm a programmer by trade.

Since I'm between clients right now, I decided to head back to the cabin to see my parents for a bit. My Dad retired in 2009 and my Mom in 2010. They started building their retirement home just up the hill from the cabin in 2004. My parents moved in two years ago and completed enough of it to show it off at a big party before Trish and my wedding last year.

The House is a majestic building, hand-built and beautifully crafted. Both the interior and exterior are amazing, with gorgeous trim and a wonderful attention to detail. The porch is possibly the best in the world, high and mighty with a great view of the cabin and garden below.

The interior is similar to the cabin I grew up in, but much more spacious. There's a wood cook stove, a heat stove on every floor and two indoor bathrooms. It was super nice to avoid walking to the outhouse in the sub-zero temperatures during my stay.

The main floor, pictured above, is warm and cozy this time of year. The upstairs, where the bedrooms are, is quite roomy and beautifully decorated. It's full of light and has a nice lanai deck facing west.

The power and water systems supplying the house are all on-demand. The Kohler generator fires up when the batteries need charging and the water pump turns on when the washing machine or dishwasher run. There's an on-demand water heater as well. Solar panel charge the batteries when the sun is shining. A good charge will power lights, internet and the TV for around 24 hours. The generator, and the backup generator, are all powered by propane.

My favorite thing about The House is everything from my childhood is still nearby. The sauna that was built in 1917, the cabin that was built the following year. There's the log barn my Dad built one winter when he couldn't find work and the Toyota Land Cruiser that we'd rooster tail down the snowy back road to get us home in the dead of winter.

More photos on Flickr &rightarrow

My parents built a whole new house and it still feels like home. On today, their 42nd anniversary, I'd like to say Well Done! You created a wonderful paradise in the mountains of Montana. I'm extremely proud of you.

Categories: FLOSS Project Planets

Bryan Pendleton: Stuff I'm reading, mid-November edition

Sat, 2014-11-15 21:46

Today we once again came to the conclusion that the middle of November is the perfect time to go to Morgan Territory Park, and that Morgan Territory Park is the perfect place to be a dog.

  • IBM X-Force Researcher Finds Significant Vulnerability in Microsoft WindowsCutting to the chase, VBScript permits in-place resizing of arrays through the command “redim preserve.” This is where the vulnerability is.

    redim preserve arrayname( newsizeinelements )

    VBScript.dll contains a runtime evaluation method, CScriptRuntime::Run(VAR *), which farms out the SafeArray redimension task to OleAut32.dll with the SafeArrayRedim(…) function. Essentially, what happens is that fairly early on, SafeArrayRedim() will swap out the old array size (element count) with the resize request. However, there is a code path where, if an error occurs, the size is not reset before returning to the calling function, VBScript!CScriptRuntime::Run().

  • Microsoft Fixes Bug That Dates Back to Windows 95According to Microsoft, the bug resides in the Windows Object Linking and Embedding (OLE) technologies that debuted in the 1990s as a way for applications to share information with each other.
  • Unintended Consequences of the Information EconomyThe DoD can’t just dial up more innovation capacity by throwing money at the problem, like they did in WWII. Nor, in a free country, can the U. S. government just mandate for whom companies choose to work. Innovation capacity requires not only brilliant engineers, who are hard enough to come by, and who cannot be easily identified in the job market, but also a willingness to accept a lot of risk: to try and perhaps to fail, over and over. To old school 20th century managers, this looks a lot like waste, but in fact it’s a necessary part of the innovation process. The economics of conflict is changing just like the economics of everything else is changing.
  • Wil Wheaton: Anonymous trolls are destroying online games. Here’s how to stop them.It’s time to break this cycle—and to teach gamers that they can compete without being competitive, that they can win and lose without spewing racist, misogynist, homophobic bile at their fellow gamers. But doing so requires casting off the cloak of anonymity.
  • Social media is brokenAnonymity can be very important for the marginalized, for whistleblowers, etc. But within their communities of trust they build reputation, including pseudonymous reputation. The real issue is feeling free of reputation, which equals feeling free of consequence. That is where bad behavior comes from.
  • Why I love databases To many of my colleagues and friends, the database is a magical black box system, too scary and complicated to understand. I wanted to change that.

    While talking about databases, the topic of distributed systems cannot be ignored. Most modern databases are distributed, either implicitly (distributed clustered databases) or externally (a single application connected to multiple databases via application-level sharding).

    This post is a confession of my love for databases and distributed systems. It is mostly targeted towards programmers like me, application developers who regularly interact with databases.

  • Climbing off the CTO ladder (Before We Fall Off)Spotify has recently started thinking about the non-management career track and we are working to see if there is a program we would like to adopt. We’ve had feedback from engineers and managers alike that we need to add a tech ladder; however, I am not convinced that adding a ladder fixes everything. The second ladder is a blunt instrument that can easily add some clarity to our growth ambitions, but it is not a complete solution. Part of management is optimizing for the productivity of the team, not our own work streams. This should not be taken lightly as it will have an effect on the organization’s culture. Understanding this may help invest in a technology ladder that works for the culture of your organization.
  • Optimizing Disk IO and Memory for Big Data Vector AnalysisFor “big data” datasets where the size of data is significantly larger than the size of memory, the most common bottleneck is disk bandwidth. Disk bandwidth of the highest-end disks remain on the order of hundreds of megabytes per second, while memory bandwidth is usually at least an order of magnitude faster. Furthermore, very little work is required of the CPU per record (just two predicate evaluations and a sum) --- database queries tend to be far less CPU-intensive than other domains (such as graphics rendering or scientific simulations). Hence, step 1 is often a bottleneck.
  • Intro to Distributed Hash Tables (DHTs)Lets assume that you want to get to some place, and you are not sure where it is. A good idea would be to ask someone how to get there. If you are very far from your destination, most likely the person you asked will give you a very vague description of how to get there. But it will get you starting in the correct direction.

    After you advance a while, you can ask somebody else. You will get another description, this time more a detailed one. You will then follow this description, until you get closer.

    Finally when you are really close, you will find someone that knows exactly where is that place you are looking for. Then your search will end.

  • Yelp Prison Review FaxbotFletcher Bach and I recently discovered that there are reviews of prisons on Yelp. Some of these reviews are snarky one-liners -for example, one yelper describes Rikers Island as a “great island getaway right in my own backyard”. Other reviews appear to be honest first person accounts. Some people review what it’s like to visit the prison; others describe their experiences as inmates.
  • Android 5.0 Lollipop, thoroughly reviewedAndroid 5.0 Lollipop is at least the biggest update since Android 4.0, and it's probably the biggest Android release ever. The update brings a complete visual overhaul of every app, with a beautiful new design language called "Material Design." Animations are everywhere, and you'd be hard-pressed to find a single pixel from 4.4 that was carried over into 5.0—Google even revamped the fonts.
  • Google's New, Improved Android Will Deliver A Unified Design LanguageAs Android lead designer Matias Duarte demoed it on stage, he explained that it moved with the physics of card stock, but also splash with your touch, like “ink rippling in a pond.” He clearly put it better than I can, though I’d add that Android‘s core UI has long been cleanly designed, but was always a bit cold. Material Design adds a bit of human warmth back to the equation.
  • Cooking the Alinea cookbookAllen Hemberger cooked his way through one of the most complex cookbooks out there, the Alinea cookbook. Aside from the chefs who work in the kitchen there, Hemberger's probably the only person to have made every single recipe. These recipes aren't easy; look at the last one he prepared...he even struggled to find the correct ingredients.
Categories: FLOSS Project Planets

Justin Mason: Links for 2014-11-15

Sat, 2014-11-15 18:58
  • IAB Statement on Internet Confidentiality

    Newly designed protocols should prefer encryption to cleartext operation. There may be exceptions to this default, but it is important to recognize that protocols do not operate in isolation.  Information leaked by one protocol can be made part of a more substantial body of information by cross-correlation of traffic observation.  There are protocols which may as a result require encryption on the Internet even when it would not be a requirement for that protocol operating in isolation. We recommend that encryption be deployed throughout the protocol stack since there is not a single place within the stack where all kinds of communication can be protected. Wow. so much for IPSec

    (tags: ipsec iab ietf snowden surveillance crypto protocols internet)

  • Facebook’s datacenter fabric

    FB goes public with its take on the Clos network-based datacenter network architecture

    (tags: networking scaling facebook clos-networks fabrics datacenters network-architecture)

Categories: FLOSS Project Planets