Photographs (from the LIFE archives) by Hugo Jaeger (Adolf Hitler’s personal photographer) The story of how LIFE came to own Jaeger’s collection of 2,000 photographs Folkert

nuclearharvest:

by YDK Morimoe

It was excellent, and for me the two highlights were hearing some of the world’s top cosmologists debate inflation theory (theirs, not ours), and Larry Page discussing  his vision for Google looking forward and why internet access by balloon makes sense.

I saw a display of Google Glass but I still don’t get it.  It struck me as excellent for people who want to send photos to their Facebook page in real time, or record their children, but that’s not me.  What I like about the iPad is that it pulls you out of the world, whereas Glass seems to integrate “the flow of information world” with “the real world.”  Why spoil two such wonderful things?  But I’ll be the first to admit that a) the defect in my understanding of Glass is my fault alone, and b) I will buy one immediately once it is available.

The best new question I heard was this: if you could change the physical laws of the universe so as to create more life in it, what would you do?  Make gravity stronger or weaker?  Change which constants?  Have stars distributed more densely throughout the universe?  More or less carbon?  And so on.  The ultimate point of the question is to get you thinking about whether our universe is fine-tuned for life after all.

The cafeteria food was not nearly as good as what I have had in the New York Google and it struck me as overrated and most likely in decline.  The vegetarian dishes were best.  What you should do is eat in the Telugu restaurant Pessaratu, Andrha mess-style food, in Sunnyvale, get the lentils and make sure you eat them with your fingers, South Indian-style, for the maximum taste experience.

Via Michael Rosenwald, here is one list of options, with attached evaluations.  I have been using Feedly and it is working fine for me.  I expected high transition costs, but within a minute or two, and then a system reboot, everything was up and running without a hitch.  I did not feel confused by the shift in visual fields, as I had been expecting.  I don’t pretend to know it is best, and I may not stick with Feedly forever, but this has not turned out to be a crisis and there is no reason to resent Google for axing their Reader.

go out and enjoy it :)

How to Carry A Plant

In the early 1900s, photographer George Lawrence built what he called the Lawrence Captive Airship, a series of kites and wires able to hold aloft a camera mounted on a stabilizing mechanism. With his invention, he was able to take aerial photos of cities all over the country -- Brooklyn, Atlantic City, Chicago, Kansas City. You've probably seen his photos of a burnt-out San Francisco after the 1906 earthquake.

You gotta love anyone whose company motto was "the hitherto impossible in photography is our specialty". (via coudal)

Tags: George Lawrence   photography

Read 5 remaining paragraphs | Comments

Toronto's growing survivalist movement strikes a balance between sustainable living, wilderness survival, and crisis preparation.

Three times each week, Joe McCumber leads up to 40 people of all ages, backgrounds, and motivations through the types of activities you might encounter at a particularly advanced summer camp: orienteering, tracking wildlife, improvising shelter, storing food, constructing a solar-powered water still. McCumber is the founder of the United Survivalist Network of Ontario (USNO), and he’s one of a growing number of people in the GTA who are spending their evenings and weekends sharing the knowledge and skills of basic survival.

Thanks largely to sensationalist media coverage, the “survivalist” label has come to suggest nervous hoarders and fringe-dwelling doomsayers anticipating a future in some post-apocalyptic abyss. The less dramatic, albeit more human, reality is that the growing appeal of sustainable living, and the advent of social networking sites like Facebook and Meetup, have made survivalism enticing and accessible to everyday people. These survivalists are interested mainly in experiencing nature and learning traditional life skills.

The Toronto Survivalism Group (TSG) has accumulated nearly 900 members on Meetup, with a mandate to teach self-sufficiency and sustainability. This is the modern iteration of survivalism, one that has more in common with the environmentally conscious, organic-food-eating, DIY ethics of downtown hipsterdom than it does with doomsday preparation.

At events around the city, TSG’s followers learn the kind of homesteading skills that, at one time in Canada’s history, were common: canning, breadmaking, natural health, and nutrition. On June 22, members are invited to an educational walk through High Park, during which author and wild-food educator Karen Stephenson will explain how to identify and prepare edible wild plants. Like many surivalism lessons, Stephenson’s teachings are equally applicable to crisis situations and basic healthy, affordable living.

For McCumber, whose workshops are regularly promoted by TSG, creating a survivalist group was a chance to impart the practical wisdom he’s gained during a lifetime as a woodsman and hunter. Granted, those skills do attract some of the more stereotypical survivalists, the ones preparing for some cataclysmic global event. And the USNO’s regular workshops on martial arts, bow-and-arrow making, and defence against “knife and weapon attacks” certainly suggest a more aggressive form of survival. But McCumber, whose pupils range in age from four to 65, maintains that general wilderness survival is his focus.

“[USNO] are not so much into the doomsday-type things, although a lot of our members do believe in that,” he said. “And wilderness skills are a good place to start if you’re into that kind of scenario.”

Though diverse in their interests, the survivalists McCumber has encountered have been able to coexist quite happily. While he admits that “there are a few bad apples in every barrel,” the feedback he has received from USNO participants has been overwhelmingly positive.

Now more than ever, Toronto’s survivalist community is deserving of a more temperate reputation. With a focus that’s equal parts healthy living, craftwork, and wilderness endurance, and with a membership that includes parents and children, high rise dwellers, and dedicated outdoors enthusiasts alike, they’re as diverse as the city itself.

CBC News has learned that a third man was at the scene of a fatal shooting in March — a shooting that has been linked to a wide-ranging police bust in Toronto. That man has not been charged in connection to the slaying.

A Mississauga, Ont., mother says she doesn't need to wait for the results of a Transport Canada investigation. She says her backyard was 'bombed' by poo from the sky earlier this week.

Read 8 remaining paragraphs | Comments

According to a video reportedly obtained by the National Enquirer, Food Network star Paula Deen confessed that she had used the N-word, told racist jokes, and wanted to hire black waiters to play the part of slaves in a wedding she was planning.

A fascinating discussion of mid-19th century women who used their opportunity to pose for daguerreotypes and the first photographs to show themselves breastfeeding.

And this is a great collection of historical breastfeeding images. Really cool stuff.

Genie is out of the bottle

by Sriram Krishnan

In a prior tech blog, we had discussed the architecture of our petabyte-scale data warehouse in the cloud. Salient features of our architecture include the use of Amazon’s Simple Storage Service (S3) as our "source of truth", leveraging the elasticity of the cloud to run multiple dynamically resizable Hadoop clusters to support various workloads, and our horizontally scalable Hadoop Platform as a Service called Genie.

Today, we are pleased to announce that Genie is now open source, and available to the public from the Netflix OSS GitHub site.

What is Genie?

Genie provides job and resource management for the Hadoop ecosystem in the cloud. From the perspective of the end-user, Genie abstracts away the physical details of various (potentially transient) Hadoop resources in the cloud, and provides a REST-ful Execution Service to submit and monitor Hadoop, Hive and Pig jobs without having to install any Hadoop clients. And from the perspective of a Hadoop administrator, Genie provides a set of Configuration Services, which serve as a registry for clusters, and their associated Hive and Pig configurations.

Why did we build Genie?

There are two main reasons why we built Genie. Firstly, we run multiple Hadoop clusters in the cloud to support different workloads at Netflix. Some of them are launched as needed, and are hence transient - for instance, we spin up “bonus” Hadoop clusters nightly to augment our resources for ETL (extract, transform, load) processing. Others are longer running (viz. our regular “SLA” and “ad-hoc” clusters) - but may still be re-spun from time to time, since we work under the operating assumption that cloud resources may go down at any time. Users need to discover the latest incarnations of these clusters by name, or by the type of workloads that they support. In the data center, this is generally not an issue since Hadoop clusters don’t come up or go down frequently, but this is much more common in the cloud.

Secondly, end-users simply want to run their Hadoop, Hive or Pig jobs - very few of them are actually interested in launching their own clusters, or even installing all the client-side software and downloading all the configurations needed to run such jobs. This is generally true in both the data center and the cloud. A REST-ful API to run jobs opens up a wealth of opportunities, which we have exploited by building web UIs, workflow templates, and visualization tools that encapsulate all our common patterns of use.

What Genie Isn’t

Genie is not a workflow scheduler, such as Oozie. Genie’s unit of execution is a single Hadoop, Hive or Pig job. Genie doesn’t schedule or run workflows - in fact, we use an enterprise scheduler (UC4) at Netflix to run our ETL.

Genie is not a task scheduler, such as the Hadoop fair share or capacity schedulers either. We think of Genie as a resource match-maker, since it matches a job to an appropriate cluster based on the job parameters and cluster properties. If there are multiple clusters that are candidates to run a job, Genie will currently choose a cluster at random. It is possible to plug in a custom load balancer to choose a cluster more optimally - however, such a load balancer is currently not available.

Finally, Genie is not an end-to-end resource management tool - it doesn’t provision or launch clusters, and neither does it scale clusters up and down based on their utilization. However, Genie is a key complementary tool, serving as a repository of clusters, and an API for job management.

How Genie Works

The following diagram explains the core components of Genie, and its two classes of Hadoop users - administrators, and end-users.

Genie itself is built on top of the following Netflix OSS components:

• Karyon, which provides bootstrapping, runtime insights, diagnostics, and various cloud-ready hooks,
• Eureka, which provides service registration and discovery,
• Archaius, for dynamic property management in the cloud,
• Ribbon, which provides Eureka integration, and client-side load-balancing for REST-ful interprocess communication, and
• Servo, which enables exporting metrics, registering them with JMX (Java Management Extensions), and publishing them to external monitoring systems such as Amazon's CloudWatch.

Genie can be cloned from GitHub, built, and deployed into a container such as Tomcat. But it is not of much use unless someone (viz. an administrator) registers a Hadoop cluster with it. Registration of a cluster with Genie is as follows:

• Hadoop administrators first spin up a Hadoop cluster, e.g. using the EMR client API.
• They then upload the Hadoop and Hive configurations for this cluster (*-site.xml’s) to some location on S3.
• Next, the administrators use the Genie client to discover a Genie instance via Eureka, and make a REST-ful call to register a cluster configuration using a unique id, and a cluster name, along with a few other properties - e.g. that it supports “SLA” jobs, and the “prod” metastore. If they are creating a new metastore configuration, then they may also have to register a new Hive or Pig configuration with Genie.

After a cluster has been registered, Genie is now ready to grant any wish to its end-users - as long as it is to submit Hadoop jobs, Hive jobs, or Pig jobs!

End-users use the Genie client to launch and monitor Hadoop jobs. The client internally uses Eureka to discover a live Genie instance, and Ribbon to perform client-side load balancing, and to communicate REST-fully with the service. Users specify job parameters, which consist of:

• A job type, viz. Hadoop, Hive or Pig,
• Command-line arguments for the job,
• A set of file dependencies on S3 that can include scripts or UDFs (user defined functions).

Users must also tell Genie what kind of Hadoop cluster to pick. For this, they have a few choices - they can use a cluster name or a cluster ID to pin to a specific cluster, or they can use a schedule (e.g. SLA) and a metastore configuration (e.g. prod), which Genie will use to pick an appropriate cluster to run a job on.

Genie creates a new working directory for each job, stages all the dependencies (including Hadoop, Hive and Pig configurations for the chosen cluster), and then forks off a Hadoop client process from that working directory. It then returns a Genie job ID, which can be used by the clients to query for status, and also to get an output URI, which is browsable during and after job execution (see below). Users can monitor the standard output and error of the Hadoop clients, and also look at Hive and Pig client logs, if anything went wrong.

The Genie execution model is very simple - as mentioned earlier, Genie simply forks off a new process for each job from a new working directory. Other than simplicity, important benefits of this approach include isolation of jobs from each other and from Genie, and easy accessibility of standard output, error and job logs for our end-users (since they are browsable from the output URIs). We made a decision not to queue up jobs in Genie - if we had implemented a job queue, we would have had to implement a fair-share or capacity scheduler for Genie as well, which is already available at the Hadoop level. The downside of this approach is that a JVM is spawned for each job, which implies that Genie can only run a finite number of concurrent jobs on an instance, based on available memory.

Deployment at Netflix

Genie scales horizontally using ASGs (Auto-Scaling Groups) in the cloud, which helps us run several hundreds of concurrent Hadoop jobs in production at Netflix, with the help of Asgard for cloud management and deployment. We use Asgard (see screenshot below) to pick minimum, desired and maximum instances (for horizontal scalability) in multiple availability zones (for fault tolerance). For Genie server pushes, Asgard provides the concept of a “sequential ASG”, which lets us route traffic to new instances of Genie once a new ASG is launched, and turn off traffic to old instances by marking the old ASG out of service.

Using Asgard, we can also set up scaling policies to handle variable loads. The screenshot below shows a sample policy, which increases the number of Genie instances (by one) if the average number of running jobs per instance is greater than or equal to 25.

Usage at Netflix

Genie is being used in production at Netflix to run several thousands of Hadoop jobs daily, processing hundreds of terabytes of data. The screenshot below (from our internal Hadoop investigative tool, code named “Sherlock”) shows some of our clusters over a period of a few months.

The blue line shows one of our SLA clusters, while the orange line shows our main ad-hoc cluster. The red line shows another ad-hoc cluster, with a new experimental version of a fair-share scheduler. Genie was used to route jobs to one of the two ad-hoc clusters at random, and we measured the impact of the new scheduler on the second ad-hoc cluster. When we were satisfied with the performance of the new scheduler, we spun up another larger consolidated ad-hoc cluster with the new scheduler (also shown by the orange line), and all new ad-hoc Genie jobs were now routed to this latest incarnation. The two older clusters were terminated once all running jobs were finished (we call this a “red-black” push).

Summary

Even though Genie is now open source, and has been running in production at Netflix for months, it is still a work in progress. We think of the initial release as version 0. The data model for the services is fairly generic, but definitely biased towards running at Netflix, and in the cloud. We hope for community feedback and contributions to broaden its applicability, and enhance its capabilities.

We will be presenting Genie at the 2013 Hadoop Summit during our session titled “Genie - Hadoop Platform as a Service at Netflix”, and demoing Genie and other tools that are part of the Netflix Hadoop toolkit at the Netflix Booth. Please join us for the presentation, and/or feel free to stop by the booth, chat with the team, and provide feedback.

If you are interested in working on great open source software in the areas of big data and cloud computing, please take a look at jobs.netflix.com for current openings!

References

Genie OSS
Genie Wiki: Getting Started
Netflix Open Source Projects
@NetflixOSS Twitter Feed

I wish I could say something more useful about the full day hearing scheduled for Tuesday, June 25, 2013 in Toronto involving Voltage's attempt to get Teksavvy to divulge the identity of potentially thousands of its allegedly infringing subscribers. I have blogged about this extensively in the last few months and recently.

One had hoped that the main materials - e.g affidavits, transcripts if any of cross-examination, and  memoranda, would have been posted by now either by Teksavvy or CIPPIC, which has been granted intervener status to do what Teksavvy is unwilling to do, which is to oppose the motion and to protect its customers' privacy.

But I can't find anything new. The docket is rather confusing, but indicates a flurry of last minute activity including a motion by CIPPIC filed on June 19, 2013 to extend the time for it to file its motion record responding to Voltage that had been due on June 18, 2013 pursuant to the Court's order of May 13, 2013.

Presumably, the motion is still proceeding at 10:00 AM in Tuesday, June 25, 2013 at the Federal Court at 180 Queen St. W., Toronto.

Live tweets will be interesting and useful, if the Court so permits. I suggest the hashtag #Teksavvy - since the real issue here in many respects is Teksavvy's vigorous and expensive effort to date apparently directed at not taking a position on protecting its customers' privacy rights, and instead relying on a law school clinic to do the job. I have suggested several times that Teksavvy's cash flow is estimated to be between $5 and $10 million per month.

I'll update as soon as I find out - or am informed of - anything more.

There's a lot at stake on this motion, not least of which is whether a wide door will be opened to copyright trolling in Canada. It's too bad that the current holder of the keys  to this door is unwilling to do anything more than throw the keys to the door up in the air, and stand by to watch where they land.

HPK

PS #1: CIPPIC has provided me with the transcript of the cross-examination of Barry Logan of Canipre, who provided evidence concerning the alleged IP addresses of the alleged infringers. For the historically minded, here's a link to the corresponding document in the BMG case from 2004. Hoping to get more documents tomorrow at the latest.

PS #2: Here is CIPPIC's Memorandum of Fact and Law.

PS #3: Here is Voltage's Supplementary Memorandum of Fact and Law.

it’s tough pitching a movie when you’re actually pitching a multimillion dollar version of the reason for your divorce hey gang, i will be appearing on twitch.tv at noon pst today with penny arcade and scott kurtz, playing dungeons and dragons: chronicles of mystara for an hour for capcomunity’s launch event!! could be good. it [...]

A monumental sculpture series by Chinese activist-artist Ai Weiwei is being officially unveiled Tuesday outside of Toronto's City Hall.