The Idea [2001]

I was a junior undergraduate physics major with a new mission. A year earlier I stumbled the field of neural networks after a (quite literal) random walk through my college library. Frustrated with quantum mechanics, for which I was studying, I got up and walked aimlessly around the library. Tucked away on a shelf on the third floor was a book by Jim Jubak titled In the Image of The Brain–Breaking the Barrier Between the Human Mind and Intelligent Machines. It was about a new way of computing based on the brain, something called ‘neuromorphic computing’ and the algorithms of learning synaptic connections called ‘neural networks’. Within a short time I became utterly obsessed, and by my junior year I was diving into the literature. I learned all about various algorithms like back propagation of error and programmed my own small neural networks. I searched the internet and the library for anything I could find. I took any class with “Neuro” in the title. I even took Neurobiology and Electricity and Magnetism at the same scheduled time, getting permission from my E&M professor to never show up for class and to take tests privately. After taking stock of the field, I came to some conclusions that I think most folks with a background in physics would agree with: (1) A brain is a phenomenal computing structure and (2) modern methods of digital computing are not capable of attaining the sorts of synaptic learning and integration functions that brains attain, in the energy and space budget that brains attain it. While modern integrated electronics were clearly on a path to approach the fundamental limits of atoms, super computers of the day were not powerful enough to do even a tiny fraction of what brains could do at the algorithmic level. The fundamental problem, I concluded, had to do with the separation of memory and processing, and a solution to the problem involved not computing the operations of a brain, i.e. reducing synaptic operations to information bits shuttled back and forth between ‘memory’ and ‘processing’, but rather to do what biology does and relying on physical circuits and, at least in part, on analog synaptic operations.

The fundamental problem, I concluded, had to do with the separation of memory and processing, and a solution to the problem involved not computing the operations of a brain … but rather to do what biology does and relying on more physical circuits and, at least in part, on analog synaptic operations.

I explored ideas for making artificial synapses out of ‘resistance changing elements’. I did not know at the time that what I wanted to make was called a ‘memristor’ and had been hypothesized by a circuit theorist called Leon Chua almost a decade before I was born. My idea, very much informed by what I knew at the time as a physics student, was to create a colloidal suspension of nano-particles suspended over the top metal layers of an integrated chip. When a voltage was applied across a gap between two electrodes an electric field would induce a dipole in the particles. The particles would, in turn, be attracted to the electrode gap and form a connection. This, I figured, could be used as an adaptive synapse: The application of a voltage gradient would induce the formation of particle bridges. It would not only act as a variable resistive element for synaptic integration, but it would mimic synaptic plasticity as well. I hit the library and found many examples of researchers aligning nanoparticles. In fact, there was a whole sub-field in the emerging field of nanotechnology called dielectrophoresis or DEP that was dedicated solely to moving nanoparticles around. Nobody, it appeared, was thinking of leveraging this effect to build adaptive connections for computing or synaptic learning elements. With a level of enthusiasm and excitement I find hard to convey, I poured my attention on the topic. After about six months of asking my mother Hillary Riggs “why has nobody done this?!” she responded “oh shut up and do it yourself. Let’s get a patent”.

A year later, with two patent attorneys, Kermit Lopez and Luis Ortiz, and Hillary as angel investors and partners, we filed the first Knowm patent.

Getting Real and Finding a Path [2002-2004]

Before I continue I need to make something perfectly clear. I know the difference between actually solving a problem and having an idea about solving a problem. I have encountered many people in my life who claim they have had an idea for something, and some rare souls who have gone so far as to patent those ideas. Few people ever take the idea all the way home–those are the people I truly respect. It is one thing to postulate a self-organizing system of nano-particles as a model of an adaptive neural network synapse and it is a totally different thing to reduce it to practice or even more difficult to something that is actually useful in the real-world.

The first problem I ran up against was that, electronically, there is no such thing as negative resistance. To represent a learning weight or synapse that could be both positive and negative, I found it necessary to use at least two ‘memristive’ connections. One connection could represent a positive or excitatory connection and the other could represent a negative or inhibitory activation. Sweeping some details under the rug, differential representation has many befits in electronics, so it’s not a mystery why it would make its way into memristive electronics. We filed a number of patents on this basic idea over the years (US9679242, US9099179, US7599895, US8909580, and US9104975), something we call a differential pair memristor synapse. Many groups developing memristor based neural networks today employ such a connection.

The next problem was more serious. How does one control the synaptic connections of a physical neural network if they are inherently unstable, owing to the fact that they are suspended in a liquid or are otherwise volatile? I really thought hard about this problem, and it tore at me. My colloidal particle synapse would be noisy and volatile. How could I use this as the building block of a learning network? The standard paradigm in Neural Networks consisted of a learning phase and an inference stage–the later requiring utmost stability of the synaptic weights. After a couple of years thinking about it I had a realization. A realization that I might have a path to a solution–and if I could only persist I might uncover one of Nature’s profound secrets.

My realization was that all brains are living systems. As a living system, a brain is intrinsically volatile. Without constant repair a brain will literally melt–that is, it will die. The synapses of a brain are really sketchy. They work only half the time, and when they do work they are noisy. They are constantly being made and constantly being destroyed. In short, brains (and all living systems) have to solve the same basic volatility problem that I had with my particle synapses. A closer inspection of the problem, however, reveals that solving this problem could lead one right to the heart of intelligence and perhaps something even deeper: what it means to be alive.

My argument went like this: If a brain is volatile then it must constantly repair itself. To repair is to build. If I knew how a brain repaired itself then I may also know how it builds itself. And if I knew how a brain builds itself then I would be closer to understanding how a brain really works. So I set out to solve the problem of synaptic self-repair.

My argument went like this: If a brain is volatile then it must constantly repair itself. To repair is to build. If I knew how a brain repaired itself then I may also know how it builds itself.

AHaH!

I found a potential solution to synaptic self-repair over the course of a couple years as a I transitioned from an internship at Los Alamos National Laboratory to a PhD program in electrical engineering at the University of Washington. My first discovery was with mathematics, working under the supervision of Reid Porter in the space data systems group and Garret Kenyon in the computational neuroscience group. I was searching for a neural plasticity rule that would repair the weights of a trained neural network. I trained a neural network with an (at the time state-of-the-art) algorithm called a Support Vector Machine. I then took the synaptic weights from this trained classifier and subjected them to noise and faults while I simultaneously operated various types of simple unsupervised plasticity rules. Almost all of the rules failed horribly. Some would take the weights to infinity. Others to zero. Others would oscillate. They all screwed up the learned state of the synapses and the performance would go to random choice. All of them, that is, except for one. One day I made a guess–and it actually worked. We watched in amazement as the neural network started to get better over time. When we damaged the network, so long as the damage was not too severe and sudden, it healed itself by actively rewiring its synapses–without supervision. The act of processing information was the act of repair! The attractor states of the plasticity rule acting on structured information led to active healing.

One day I made a guess–and it actually worked. We watched in amazement as the neural network started to get better over time. When we damaged the network, so long as the damage was not too severe and sudden, it healed itself by actively rewiring.

I termed the rule Anti-Hebbian and Hebbian (AHaH) plasticity, as it required a combination of both applied in a certain way. I now call this the unsupervised AHaH rule, as my subsequent exploration into AHaH plasticity has greatly expanded what is possible, now encompassing multiple areas of machine learning.

After a year at LANL I applied for graduate school and was accepted to a few institutions and rejected by a few. Nobody that I could find was doing exactly what I wanted to do, although Dan Hammerstrom’s group–then at the Oregon Graduate Institute, was close. I applied to work with him, but I did not get any financial aid, so that was sadly out of the question. When I spoke to some professors about analog circuits at the time they mostly laughed, telling me that Moores law made it unnecessary. I told them what would happen when Moores law died in the near future, and they rolled their eyes. The only place to give me financial aid was University of Washington in Seattle. I had college friends in Seattle, and I certainly did not have any money, so I went to UW with a TA position and a Boeing fellowship.

The Grueling Graduate Grind

By the end of the first quarter a few things had occurred that profoundly changed my life. While standing in line for registration I met Knowm Inc co-founder Tim Molter. One way or another we have been working together ever since. One of my first courses was digital CMOS design. I learned how modern digital computers worked at the hardware level, and how chips were built. Tim and I took a MEMS course from Dr. Babak Parviz, a funny and brilliant man who would later go on to develop Google glass. Taking the particle-synapse idea and combining with a project that Tim’s now-wife Britta was working on, we designed an immuno-assay system for measuring hormone concentrations in the scat of wild animals. It turned out that transporting poop across country borders was difficult, so on-site measurement was needed. Our idea was that you would take a little poop, mix it with water, and dip the device into it. Leaving out the technical details and jumping straight to the punch-line, we named our device the “Dielectrophoretic Scat Hormon Immuno-Assay Technique” or DEPSHIAT. That was fun. We could hardly present and keep a straight face as we dropped that acronym. I also took a robotics class and learned about the mathematics of multi-joint arm translation, which I found overly rigid and wondered why we could not just build neural networks that learned to do such things.

I was a TA for the lab portion of an undergraduate course on electrical engineering, what they informally called the “weed out class”. It was horrible. The majority of the students could not understand the material sufficiently to even ask a question. They piled into my tutoring sessions, hoping to find some glimmer of understanding–waiting for me to say something that would make sense. My official job was to just answer questions–but what do you do when they can’t even ask a question? What do you do when they all just stand there, perfectly attentive and quiet–waiting for you to say something that makes sense? Rather than turn them away, I began to make my own lectures to cover the material so I would have something to say, and the attendance to my study session grew to include students from other study sessions. Of course, this consumed even more of my time. In short order I lost any semblance of a life and was put to work instantly and intensely for a gigantic and impersonal academic machine.

While this was alone sufficient to cause depression, there was something else at work. I grew up in the country with a river below me and a mountain above–no street lights, no pavement or concrete. I was surrounded by nature in all directions and could go pretty much wherever I wanted–and I did. The occasional barbed-wire fence erected by valley farmers were easily breached. I climbed the mountains and swam and fished in the rivers. I made tree-houses, zip-lines and flew model airplanes. There were hardly any people, but I am not a particular social person anyway. I enjoy thinking and building things–I do not crave constant social interaction. The city is a whole different animal for me. I crossed the road only when the light told me, and they flashed a precise number of times. I kept to the side walk. I traveled at certain times of the day, constrained to bus schedules that varied only by minutes or even seconds. I was surrounded not by nature and endless opportunities for me to explore but rather private property, locked doors and masses of grumpy people actively avoiding eye contact. For somebody who grew up in the country, I felt like I was locked in a box. All I wanted to do was think about the AHaH circuit problem, and my circumstances would not let me. When I combined the TA stress with the claustrophobia of the city, this was the least happy I have ever been in my life.

The River Eddy[2005]

I began to withdraw from my classes and focused on my own work. I ran simulations on my PC and published a paper with my old LANL supervisors Reid and Garret. I discovered that a number of mathematical rules would accomplish the same thing as the first AHaH rule so long as they followed some simple guidelines. I suspected that I had a potential solution to my ‘volatile synapse’ problem. If I could find a way to reduce the AHaH rule to a physical circuit then I would be able to constantly heal the particle synapses. One weekend while taking a break with a group of friends at my friend Nicole’s cabin, I was standing by the river. I threw a log into the water and watched briefly as it rode the current down stream. I stared at my feet and thought about the AHaH circuit problem. A couple minutes later, the log appeared at my feet again. In confusion I tried to figure out what happened. It turned out there was a big eddy. The log had been trapped in the eddy and was brought back, a big whirlpool or–because I was thinking about circuits at the time–a positive feedback loop. Suddenly it came to me. I knew in this one instant how to get the AHaH circuit. Whereas I had previously been trying to get the rule all at once, my realization was that it was a temporal process and that I had to decompose the circuit into distinct temporal steps or stages. I called it the “flip-lock cycle”. Perhaps because I was making such a fuss, Nicole took this picture, which I’ve illustrated. The rubber-ducky hat is regrettable.

Suddenly it came to me. I knew in this one instant how to get the AHaH circuit.

The whole drive back to Seattle I sat in the back seat drawing diagrams in my notebook, utterly absorbed. A few of these sketches made it into patents, which I filed thanks to my KnowmTech partners. Perhaps in the following diagram you can see how my mind went from the river to the circuit:

Dropping Out

While at UW, my first patent issued and I got an invitation to speak in Silicon Valley. My advisor had been promoted to dean and I was being passed off to other professors who had no intention of supporting my rather specific interests. I had the choice of pursuing graduate work with fire-fly chemistry on a chip as a route to optical signaling or I could drop out and do my own thing. At the same time my father’s health was taking a dramatic turn for the worse and he had just lost his home, where I grew up, to the bank. It was clear he might not live much longer. I made a list of all the pros and cons to staying in graduate school or dropping out. It was the hardest decision I have ever had to make, and it’s a decision that still haunts me. I dropped out of school to be closer to my father as he died and to try to build a brain.

A university dropout wanting to build a brain with no connections, no money, and no influential family members has almost no options. I packed up my stuff and moved back to New Mexico. I did not have the equipment or money to fabricate a synapse and build an AHaH circuit. All I had was my own brain, a PC, and a brief window of time. So I did the only thing I could do: I thought about it. In a few months I had worked out that unsupervised AHaH attractor states operating on binary patterns were universal logic functions and that they could be used as the basis of a new type of computing. I explored large pattern recognition architectures, performed some simulations (limited to the resources of my PC), and filed more patents with my partners. One of the biggest realizations I had during this time was that the liquid particle device was just one example of what could become a massive universe of possible devices. I asked myself how I could think about adaptive resistive devices more abstractly. I settled on the idea of a ‘collection of meta-stable switches‘ and we filed more patents.

By June of 2005 I had developed the core of what would become known as “AHaH Computing”. With the help of Knowmtech partners, particularly Hillary, we participated with a small booth at the Semicon trade show in San Fransisco. We did not go to sell anything to anybody. We just wanted to meet people and introduce the idea to the world. Some laughed, some were intrigued, and many just ignored us. One man by the name of Henry Becker recommended I speak with some people in DC and helped arrange the introductions.

Going Into Debt

In December of 2005, Hillary and I, using a line of credit to pay for it, made a trip to DC to follow up with the introductions provided by Henri Becker. The primary focus of the trip was to present for the Atlantic Nano Forum at the US patent office.

The reception to the talk was generally good, and one woman even asked for my autograph!

We met with the Office of Naval Research. We got a brief tour of the facilities and then sat in a conference room with about half a dozen ONR engineers. Barely two minutes into explaining the idea I was interrupted and given a lecture about how things should be. The objection was irrelevant–it was based on an assumption that was incorrect–but I could not break through. The meeting descended into a conversation with the other engineers in the room, talking as if I were not there, about a problem that did not exist.

Next we met with a guy at the National Science Foundation. We were told by a man who’s name I will never forget that “we don’t fund science projects” and “everybody knows electronics and liquids do not mix” and to “get out of my office”. I remember coming back to the hotel after the NSF meeting and just going to sleep instead of eating dinner. When you have so much invested, being rejected so harshly and arrogantly is incredibly disappointing. I learned much later that Alex Wissner Gross demonstrated the basics of the DEP-particle idea using NSF facilities the very next year after our visit to NSF. In an ironic twist, last year the NSF paid my plane ticket to Hawaii for the Thermodynamic Computing workshop.

I have learned many times since then that research funding decisions are political affairs, and that inside connections and positive brand recognition are extremely important.

I was told by the National Science Foundation that “we don’t fund science projects” .

Our last day in DC, after getting turned around in the endless beltways and arriving late, we met with Todd Hylton, then director of the nanotechnology division of SAIC. I remember walking into the massive shiny building, going through security and riding the elevator up to his office. After hearing what I had to say and even asking a few non-judgmental questions, he told me “you are either a genius or your crazy and I do not know which one. If you help me understand this, we might be able to work together”. Unlike the others who simply assumed they knew what I was saying and that I was crazy, Todd listened and asked questions to make sure he understood. It probably helped that he had a background in physics but otherwise had no prior experience in the fields neuromorphic or machine learning to bias his judgment. I promised to put together a tutorial on a website for him and others and the next day Hillary and I left DC in a snow flurry for a meeting at Albany Nanotech in New York.

Todd generously used his Christmas break to review the material we provided, and soon thereafter laid out a plan for a research collaboration with SAIC, which is summarized fairly well in an email from that time:

Tasks:
1) Todd to call Michael Huff about collaborations
2) Ed to forward the Lacoste BAA to Alex and Hillary.
3) Todd to send Alex a list of names to add for tutorial access.
4) Alex and Hillary to supply Todd with information on applications. Hillary to develop a narrative and visuals describing the value of the KnowmTech technology in a selected application.
5) Alex to provide Todd with information on competitive technologies including other neural network technologies and competition from other technologies like DSP.

Suggested seedling program plan task items:
1) Fabrication of a crossbar chip (possibly by MEMS Exchange)
2) Acquisition of candidate nanoparticle systems (and development of suspension chemistries if needed)
3) Characterizing the physics of the nanoparticle/electrode interactions.
4) Modeling and simulation of the synapse performance
5) Construction of a breadboard device to including board level electronics attached to a crossbar chip with nanoparticle suspension capable of testing ~50 synapses.
6) Comparison of a breadboard device performance with modeled results and extrapolation to large system performance

In the year following our DC trip, I built the website tutorial and disclosed what I had figured out up until that point, gathering all the supporting evidence. The Navy and NSF reception to my ideas made it clear that I would not be able to pitch my ideas to get research funding directly. I needed an insider–somebody with a brand. I did not have a PhD, I did not work for a big prime defense contractor, I was too young (25 at the time), and I had absolutely no connections to anybody with any relevance. Todd had all these things–he could walk into those very same places and present the exact same information and come away with a contract instead of ridicule.

Todd decided to pitch the idea to DARPA as a ‘seedling’, which is a small ~$750k program that is used to demonstrate the feasibility of an idea. The program would be under SAIC and I would be a sub-contractor. $750k sounded like a massive amount of money to me at that time, and I had yet to learn about just how little would be left over after SAIC took its bite. Ignorance is bliss, as they say, and something is a whole lot better than nothing! We worked on the presentation for months and after eleven versions Todd was ready to make the pitch.

I was nervous but excited. Suddenly, only days before the pitch was scheduled, in early 2007, we got some very bad news. SAIC was shutting down the nanotechnology division and Todd was out of a job. Unfortunately, there would be no program. Todd had three options. He had a job offer at Johns Hopkins, he had interview with Lockheed Martin, and he had been approached by DARPA. Todd accepted the Johns Hopkins offer.

Suddenly, only days before the pitch was scheduled, I get some very bad news. SAIC was shutting down the nanotechnology division. Todd was out of a job. Unfortunately, there would be no program.

As for me, I was at the end. There was really nothing else I could do. Hillary and I were going into debt and living on fumes. One day it was all ready to go and the next it was over, the hope dissolving over night. I started looking for and interviewing for programming jobs in and around Santa Fe. Thinking about my decision to drop out of graduate school I realized I had made a terrible and perhaps unrecoverable mistake. Elon Musk has said that ‘being an entrepreneur is like eating glass and staring into the abyss of death’. I think I know exactly what he is talking about.

Being an entrepreneur is like eating glass and staring into the abyss of death.
-Elon Musk

DARPA SyNAPSE[2007-2011]

Having already accepted the job with Johns Hopkins, Todd nevertheless kept his job interview at DARPA. He asked if he could use our presentation. Rather than go representing SAIC, he would go representing himself and he would use our presentation as an example of the sort of work he would do as a program manager. As luck would have it, then DARPA director Tony Tether had just been given a scolding by congress for ‘not spending enough money on brainy people’. So after Todd finished pitching our idea for DAPRA to spend a bunch of money on brainy people to build brains with nanotechnology–director Tether exclaimed “this is exactly what we need to be doing” and hired Todd on the spot. Todd called with the good news from DARAPA: we had our seedling program! My guess is that Johns Hopkins was not amused.

In the space of two days I went from feelings of crushing failure to immense excitement and relief, although I was mostly still in shock. This experience taught me a few things. First, a situation can change dramatically, for the better or worse, literally overnight. Second, sometimes you have no control and yet, somehow, things work out.

The feeling of success did not last too long. Todd called me and said that the program idea was considered too developed for a seedling and that it would go straight to program. A big program. I was actually excited to see my work taken up so quickly, but Hillary knew instantly what this meant–and it was not really good for us. It took me longer to catch on, as I was still rather young and naive about the ways of the beltway bandits. Todd used the presentation to get hired, and DARPA used the presentation to justify a large program that would fund much larger organizations. There was no way I could compete for a fifty million dollar DARPA program–such programs are won only by the largest and most well connected government contractors–companies that were now going to be paid many millions of dollars to solve the problem I came to DC to find funding for.

I was actually somewhat excited to see my work taken up so quickly, but Hillary knew instantly what this meant–and it was not good. It took me longer to catch on, as I was still rather young and naive. We had been by-passed.

While Todd probably could have given me a seedling to fund my work, he opted instead to hire me on as a Science and Engineering Technical Advisor (SETA). I did not have a choice one way or the other, and I’m honestly not sure what would have been better. In May of 2008 Stan Williams of HP made the now infamous announcement that they had “discovered the memristor”, the “missing fundamental circuit element” and on November 5th later that year, with the backdrop of Barack Obama being elected present, the DARPA Systems of Neuromorphic Adaptive Plastic Scalable Electronics (SyNAPSE) program kicked off in the ballroom of the Dolce Hayes Mansion in San Jose CA. HP, IBM and HRL were selected from about two-dozen proposals from all the top DoD prime contractors. I would spend the next four years on the government advisory team watching it all unfold. In 2011 Todd left DARPA to work at Brain Corp with Eugene Izhikevich– formerly on HRL’s SyNAPSE team. I first met Robinson Pino, then at Air Force Research Labs, while on site reviews for the SyNAPSE program. He informed me of an STTR solicitation that was very close to my work. With only 10 days left before the proposal was due I rushed a proposal–and won it. I left to pursue one of a few small programs over the next five years to further develop the Knowm technology stack. My thinking at the time was nicely captured in a practice presentation for the program kickoff.