This paper was detailing the approach taken by the winning team of the DARPA Grand Challenge, a robotics competition to build an autonomous vehicle to drive over 100 miles across desert terrain.

The architecture used in was of interest to me. Specifically the “three-layered architecture” which “pipelines data through a series of layers, transforming sensor data into internal models, abstract plans, and concrete robot controls.” I intend to research three-layered architecture in more depth to better understand the specifics and the motivation for this particular architecture.

The authors make extensive use of something called an unscented Kalman filter (UKF) for determining drivable areas from laser and other data. I’m not familiar with this and need to do more research to better understand it. They use the UKF to build a model of the environment and they utilize a Markov model to model the noise in the UKF model over time. The Markov model degrades over time due to the increased potential of error drift from introduced false positives. To fix the degradation over time they used a “discriminative machine learning algorithm”.

The lasers used had a limited range and therefore safe operating speeds were limited to 25mph. To extend the range of the model on drivable space, a vision system was incorporated into the existing model by adding in the difference in gradient surface color. This effectively extends the drivable range to a point where the safe drivable mph can be increased to 35mph from 25mph.

They also created a velocity control system that detected shocks created from the terrain. This allowed for slow acceleration to maximum velocity as well as a reduction in velocity during turns and other scenarios requiring it. They used a human driver to train the acceleration parameters in order to mimic human velocity control during driving. They highlight the adaptive velocity control as essential to their victory over other competitors especially since they were the only team to have such a system.

Winning the DARPA Grand Challenge with an AI Robot. Michael Montemerlo , Sebastian Thrun , Hendrik Dahlkamp , David Stavens. In Proceedings of the AAAI National Conference on Artificial Intelligence

The authors present their research on matrix factorization with respect to their winning entry in the Netflix Prize for a recommender system.

One point of interest or clarification the authors made on my understanding of matrix factorization was that extrapolating characteristics from patterns within a dataset is called “latent factor modeling”. Latent factor modeling is an approach to infer patterns from a database that define characteristics of the data. Similarly, a human could manually define characteristics of a movie which could be of importance: actors, genre, length, theme, etc. Latent factor modeling is an algorithmic approach that infers those characteristics from patterns in the data without knowing precisely what each characteristic is. In this case, latent factor model builds a matrix of users and items (movies) with each point being a vector of characteristics.

Matrix factorization allows for a recommendation to be made by taking a sparse matrix of ratings and taking the dot product with the latent factor model. Specifically, matrix factorization is the factorization of the latent factor model of n users and m items with a sparse matrix of ratings each user has made on some items. The authors also present various extensions that helped them win including bias, additional input source, temporal dynamics, and confidence levels.

The interesting part of this paper at this point for me is the “latent factor modeling”. I intend to research this a bit more to understand how the model is created.

Matrix Factorization Techniques for Recommender Systems

In 2006, Netflix announced a 1 million dollar prize for an algorithm to recommend movies to users that they would be interested in that could beat their current algorithm by at least 10%. This paper reviews the methodology taken by the winning team to achieve victory after 3 years of work.

The winning solution was a combination of methods incorporating weighted nearest neighbors, matrix factorization, gradient decent, temporal drift, and various extensions to the standard methods employed with each technique. Numerous models of the data were created using these methods and the winning solution was an ensemble, or averaging, of the models. The winning team was a collaborative effort among many researchers and the authors highlight the benefit of croudsourcing to finding a winning algorithm.

The part of the algorithm that I liked most was matrix factorization. Its a technique that can extract meaningful relationships from an unknown dataset. In this case it was able to separate funny movies from serious movies, dramas from comedies, and so on. The utility of such a method can be easy extrapolated to a vast number of other problems. I plan to investigate matrix factorization in more depth as a result.

All Together Now: A Perspective on the NETFLIX PRIZE. Robert M. Bell, Yehuda Koren, and Chris Volinsky

This is a brief review the introduction article for the Machine Learning Journal “Special Issue on Empirical Evaluations in Reinforcement Learning”, Volume 84, Numbers 1-2 / July 2011. The article does a good job at tying together seemingly disparate empirical research papers on machine learning. It presents a few past empirical studies and provides brief summaries for each of the papers presented in the current issue of the journal. Topics include:

• How should software for empirical research be designed?
• What are the right evaluation methodologies?
• What are suitable benchmark tasks?
• How do current methods fare in empirical comparisons?

While no definitive answers to any of the questions is presented, the papers summarized by the author’s of this introduction highlight current progress towards answering them from an empirical research perspective. Overall, this was a well organized paper that brought with it the depth of the author’s understanding of the field of machine learning. It puts the unfamiliar researcher on firm footing for further exploration of the state of the art in machine learning.

Introduction to the special issue on empirical evaluations in reinforcement learning

This site has been a long time coming. I pretty much finished the prototype about 6 months ago and the CEO is working on funding to take it to the next level. Its a great way to easily contribute to charities around the world without costing you a single penny. It basically takes profits collected per transaction and donates them to your selected charities or causes. By integrating into your existing shopping cart, you can help save the world without costing you a single penny more than what you’d already be paying in processing fees. Take a look at the site and if you’d like more information please don’t hesitate to contact me or one of the other contacts on the site.

I used to brush aside UJS as unnecessary and a source of maintenance problems because of the disconnect from the HTML. Developers coming in with no knowledge of the JS will have problems debugging complex JS that’s been unobtrusively created. I do still feel this is true but the benefit of UJS is in the simplicity and elegance of your JS code in the end. It might add a layer of complexity because of the detatchment from the HTML but the JS you end up developing will be cleaner, more concise, and ultimately much much more readable than embedding it into the HTML.

With that rant out of the way, lets examine a problem with the Prototype library. When doing Ajax calls we can set onSuccess, onFailure, and a few other observers. With UJS your Ajax observers might look something like this:


$('something').observe('ajax:success', function(right_evt) {
alert('success');
});
$('something').observe('ajax:failure', function(right_evt) {
alert('success');
});


But what if our Rails app makes smart use of HTTP status codes and returns status codes like 401 or 302? With Prototype, we can specify on401 or on302 but the catch all observers like onFailure won’t get called in those cases. For instance, just adding the following to the above code:


$('something').observe('ajax:on401', function(right_evt) {
alert('unauthenticated');
});


Has 2 problems. First, it might seem like this is possible with Prototype the Rails UJS, rails.js, doesn’t actually recognize any specific status code observers. So that on401 will just get ignored. There are 2 solutions to this. The first is to hack up rails.js and add a listener for on401 or whatever other status codes you want to specify. The problem is Prototype will ignore the other catch all observers like onFailure if you start specifying specific status codes. A better solution is to just catch the specific status codes that you want custom inside of the catch all onFailure observers. You can do that like this:


$('something').observe('ajax:failure', function(evt) {
if(evt.memo.status == 401) {
alert('unauthenticated');
return false;
};
alert('Failure');
});


Now this will handle 401s differently than other failures but still catch all the other failures without having to specify all of the status codes individually.

Summary:
This paper presents a review of current brain-machine interface research with respect to how such BMI devices would be used for neuroprosthetic devices. The authors present a high level overview of the current research at each step for a neuroprothestic device to be realized. There are currently 4 main techniques for reading data from the brain: single-unit sensors, local field potentials, ECoG, and EEG. Single-unit sensors consist of tiny sensors places directly next to the neurons they are reading. Local field potentials consist of a small array of sensors located within a micro region of interest and can pick up signals from multiple neurons. It uses the local field potentials of the nearby neurons to determine if any of them are firing. ECoG is embedded into the skull or directly on the outer cranium and picks up electrical fields over much larger areas than the other two techniques. EEG is usually non-invasive and is the least accurate as far as detection of single neuron firing. Each method has its pros and cons usually with relation to invasiveness versus signal precision.

After signals are processed, they need to be interpreted and turned into actions. The ability to decode actionable intentions from the sensors is an area which requires more work. After identifying and translating thoughts into actions, it is important that neuroprosthetic devices contain feedback mechanisms. The authors discuss a handful of different feedback techniques all of which are still actively being researched. There is no best method for providing feedback and any quality BMI neuroprosthetic device would likely contain multiple feedback methods.

Questions:
This paper is 2 years old and I’m curious how much the state of the art has changed since its publication. I’ve read of what seem like a few advances in the areas presented by this review but nothing conclusive to say a key problem has been solved. For instance a recent DARPA funded neuroprosthetic device in humans is already in trials. Also, recent theoretical work to understanding the Connectome is still many years away but the authors seemed to brush over the concept of understanding the complexity of the neural connections themselves in order to properly identify neurons of interest.

Future Work:
This paper was a review paper and was pretty much focused on presenting the future work to produce neuroprothestic devices. Almost everything in this paper could be considered future work. One area that particularly interests me is the need for more research to understand various feedback mechanisms. Current work with feedback other than visual feedback is limited.

Another area of work that seems interesting is the need to develop learning algorithms that can improve operation with feedback from the devices as well as adaption to the subject’s neurons. Adaptive learning algorithms must be developed which are robust enough to work in dynamic environment with neural connections having the ability to rewire themselves. My prior experience with genetic algorithms may provide a possible research transition into neuroscience with respect to signal processing, feedback interpretation, and actionable intent identification using genetic algorithms as a part of the learning algorithm.

Conclusions, Interests and Reflections:
This was a great paper for me because I lack much depth in the field still and it provided me with a really good overview of the state of the art in neuroprothestics. I was conducting my own review of the field so this was a great crash course on BMI neuroprosthetic devices, albeit, a high level overview. It put things in perspective and outlined several potential future research paths that I may follow.

Parag G. Patil, Dennis A. Turner – The Development of Brain-Machine Interface Neuroprosthetic Devices – Neurotherapeutics – January 2008 (Vol. 5, Issue 1, Pages 137-146, DOI: 10.1016/j.nurt.2007.11.002)

Summary:
This paper presents the mathematical formulation for handling parallel RF transmission inside the magnetic field of an MRI. It gives mathematical evidence of the possibility of preserving image reconstruction using the parallel RF transmissions. They provide experimental evidence showing a prototype of their mathematical formulation in action.

Conclusions:
This paper makes it clear to me that there are still gaping holes in the depth of my knowledge of MRI technology. I need to fill those gaps so that I can really understand the details from the magnetic field generation to gathering data and the algorithms for image reconstruction. There’s way too many details that are missing in my knowledge for this paper to be much use to me mathematically. It does, however, present the idea that RF transmission is possible in conjunction with MRI and that the algorithms can be modified to account for the RF transmissions. Without spending much time mulling over the mathematical details, I can definitely postulate that ‘if’ we were able to use RF transmissions for remotely firing neurons that we could include those transmissions mathematically in the image reconstruction for the MRI images and not have the transmissions interfere with the imaging. If we were to construct a feedback loop using RF to transmit to the brain remotely and MRI to read normal brain activity as a response we could theoretically create a continuous scan without having the excitation RF interfering with the normal MRI function.

Ulrich Katscher and Peter Bornert – Parallel RF transmission in MRI – NMR Biomed. 2006; 19: 393–400

Summary:
This paper studies the use of low intensity, low frequency ultrasound to elicit neuron firing. The authors test their approach on a 2d culture of neurons taken from mice brains. They subjected the culture to ultrasound (US) at varying frequencies, intensities, repetitions, and durations. They analyzed the cultures’ response to the US by looking at Na and Ca quantities within the culture. Its understood that increases in Ca and Na relate to neuron firing. The experiments show that US was able to excite neurons into firing.

They also tested their approach on an entire extracted mouse brain to see if the US could penetrate deep into the brain tissue and have the same effect. They observed the opposite side of the brain that was being subjected to the US which showed neuron activity.

The authors cite mention research that suggests US has negative long term effects on neurons making the use of US for continual exposure potentially dangerous but they find no evidence of disruption in their cultured neurons.

Questions:
I was extremely interested in their testing of excitation through the entire brain, however, the approach doesn’t seem to present a way to excite a specific depth within the brain. The authors seemed to only test the firing of the far side of the brain area being stimulated. The paper seems to avoid the question of whether the US would effect firing of all neurons within the path of the US. Its likely they would require a more precise neuron firing detection method to analyze firings at depth. Use of fMRI would likely make sense for analyzing neuron firing in 3d but the ultrasound equipment would likely interfere with the fMRI scan without proper integration into the fMRI scanner.

Future Work:
One potential area for future work would be to investigate long term effects of US use at varying intensities frequencies. It is of particular interest to develop technologies to elicit neuron firing that can be used continuously for extended periods of time. It is extremely important to understand the long term effects of US at different intensities and frequencies so that we can potentially identify frequencies and intensities which minimize the negative effect of using US to fire neurons.

Another area of potential research would be attempting to use US in eliciting targeted neuron firing within 3d space. In particular, the firing of neurons deep within the brain without effecting the surrounding areas. The current research shows techniques on a mostly 2d plane of neuron culture and a general analysis of an entire brain but fails to show how US would be capable of targeting only neurons at specific depths within the brain rather than every neuron along the US path.

Conclusions, Interests and Reflections:
The authors conclude US is suitable for neuron firing which is true but the negative prolonged effects may outweigh the benefit of US neuron excitation over other methods. I was extremely interested to see evidence of precise neuron firing in the 3d space with their entire brain experiment but it seemed missing. The authors were focused on showing that US does elicit neuron firing and how it works. It was an interesting approach but the negative effects of US on neurons over time will be the biggest drawback to the adoption of this method. The authors found no evidence of neuron disruption but cite other research which does. Without identifying safe frequency and intensities at which to operate over long periods of time, the US usefulness for neuron excitation may be limited to procedures requiring minimal temporal US stimulation.

William J. Tyler, Yusuf Tufail, Michael Finsterwald, Monica L. Tauchmann, Emily J. Olson, Cassondra Majestic – Remote Excitation of Neuronal Circuits Using Low-Intensity, Low-Frequency Ultrasound – PLoS ONE 3(10): e3511 (2008).

Summary:

This paper describes a technique for using RF magnetic fields to heat an aqueous solution of nanoparticles bound to fluorophores within a culture of genetically engineered neurons to fire individual neurons. The fluorophores act as a thermometer by lighting as the nanoparticles are heated. They experimentally measured the temperature of the bulk solution without the cells as they target a local nanoparticles with RF magnetic fields to heat them locally. They concluded that the nanoparticle heating was concentrated locally and didn’t result in an increase in temperature for the bulk solution. The temperature of the local heated nanopartical did not radiate enough away from the target of the RF magnetic field to effect the bulk temperature. They genetically engineered neurons to contain the fluorophores they tested in the bulk solution so that the nanoparticle binds to it.

Applying the RF to the nanoparticles heated the cells locally and they measured the calcium concentration using a calcium sensor and voltage using voltage sensitive dye within the culture showing that it increases meaning the neurons have fired.

They provide experimental evidence of initiating a physical response in worms. They tested heating nanoparticles injected into the worms motion control area and when RF was applied the worms reversed their forward movements. RF waves without nanoparticles had no effect. This experiement showed that the RF magnetic field did heat the nanoparticles and subsequently triggered neurons to fire within the worm.

Questions:

Why was this type of worm selected? Does it have specific characteristics that were ideal for experimentation of this kind?

What other nanomaterials behave similarly?

How locally targeted was the RF field? Early in the paper its suggested the RF field was applied locally to observe heat difussion in an aqueous solution of nanoparticles but later in the paper its mentioned that RF fields have a problem targeted specific cells. I’m confused. Is ‘locally’ a relative term with respect to the medium? Local in the bulk solution meaning an inch which seems reasonable if a solution is 12 inches in diameter but an inch with respect to neurons in the brain is enormous.

Future Work:

The article mentions difficulty in targeting specific cells for heating using RF magnetic fields. The ability to more precisely target specific cells would open the possibilities for neuroscience research immensely. An important area of research is more precise targeting of RF magnetic fields. This might be an intractable problem, however, with RF fields in general. Perhaps if a nanoparticle would heat only when a magnetic resonance of a certain threshold is created we could then potentially use existing MRI technology to both image the brain and elicit neural firings at a higher spacial resolution than the RF waves themselves can provide. Research into different types of nanoparticles which are heated only when magnetic resonance is within certain ranges and no heating otherwise would be ideal. My knowledge of magnetic resonance isn’t deep enough to know if this is even feasible however. Some research currently uses fiber optics inserted into the brain to isolate individual neurons for firing. I’m not sure if they use light or heat but I think its light. If photons of light can trigger neurons to fire, it seems reasonable that the photons emitted from MR could potentially either directly or indirectly fire neurons as well.

One area of future work would be applying this technique in more complex organisms. In addition, it would be extremely useful if the nanoparticles could be intravenously given to the organism and effectively expand response potential over the entire brain. Effects of the nanoparticles themselves on normal brain function and other biological function would have to be thoroughly evaluated.

Conclusions, Interests and Reflections:

I read a review of this article which seemed to hype up the results. While extremely interesting, it goes without saying that we are a little ways away from being able to apply this type of method to a human brain to target individual neurons anywhere in the brain. This research shows the potential of RF magnetic fields and nanoparticles to expand the possibilities of non-invasively interacting with neurons.

Heng Huang, Savas Delikanli, Hao Zeng, Denise M. Ferkey and Arnd Pralle – Remote control of ion channels and neurons through magnetic-field heating of nanoparticles – Nature Nanotechnology Volume: 5, (2010)