> We’re still a young company early in our mission...

What? Why don't they consider themselves part of Google?

I don't know why, but it's very clear they consider themselves somewhat independent of Google.

The most clear sign I have seen is that Partnership on AI site has both Google logo and DeepMind logo.

https://www.partnershiponai.org/

Probably by design.

I assume Google wants to have them doing their own thing instead of just borg-similating it. Also they have Google brain, which is doing similar things

Why not just use GNU Parallel (or something similar) instead of Spark?

I think this could have been done with GNU parallel. One advantage I see with Spark is that is that it is easier to interact with Python, for example these two lines are all is needed to call the relevant Python function:

  urls = sc.parallelize(batched_data)
  labelled_images = urls.flatMap(apply_batch)

So if you already have a cluster with Spark installed (like Databrick does) then it takes less work to just call your Python code than setting up a GNU Parallel cluster and a writing a small wrapper script. Additionally a Python script would have to load/init the models on every call from Parallel. I agree that this is not a great demonstration of Spark main strengths.

I think one reason would fault tolerance. Is there a fault tolerance layer in GNU parallel? last time I checked their homepage ( a few minutes ago), there was no reference to fault tolerance.

Another reason is, perhaps, scheduling.

what fault tolerance does spark give you in this scheme? It cannot look into TF progress and checkpoint all state. Using Spark with TF, seems like an overkill -- you need to manage and install two framework what should ideally be a 200 line python wrapper or small mesos framework at most.

Does --retries count as fault tolerance?

Spearmint, for example: https://github.com/JasperSnoek/spearmint/blob/master/spearmi...

I think you mean https://www.whetlab.com/

I agree these methods still require a fair amount of expert knowledge and intuition in order to make the various choices you mention. On the other hand, Bayesian optimization can prove useful for exploring such a space. A recent paper (http://arxiv.org/pdf/1206.2944.pdf) used Bayesian optimization with GPs to find hyperparameter settings for a deep convolutional network. The resulting hyperparameters gave state of the art performance, beating those chosen by an expert (Krizhevsky, the researcher who recently won ImageNet).

Agreed. Here's a previous HN discussion on that topic: http://news.ycombinator.com/item?id=4281630

Given that pharma is a massive industry and that drug discovery often costs around 1 billion dollars, the top prize of $22,000 seems awfully low. Will we start to see larger prizes, or will startups take this technology and monetize better than academia currently does?

With Geoffrey Hinton involved as a supervisor I expect they were on the bleeding edge for other reasons anyway and just decided to scoop up some extra cash as well. I've not looked closely but Kaggle does seem to be a little like 99designs though.

The Heritage Health Prize is $ 3 million! No exactly 99designs regime. [1] https://www.heritagehealthprize.com/c/hhp

The heritage prize is $3 million dollars if a certain threshold score is met.

I'm on the team currently in 1st place, and I don't think there is any chance that any team will meet this threshold.

So, the final prize will be $500,000. Still, not 99designs.

Interesting. Thanks for the clarification!

There is no shortage of people interested in machine learning.