Local versus Global Optimality

Neural Nets & Evolutionary Algorithms can be pretty dumb & pretty smart at the same time. How? The same way that quantum physics works where we have local conditions that work for our immediate locale/environment & quantum or global conditions  on a massive scale in the rest of the universe that may be invisible to us.

When you give an adaptive AI machine a learning set, it self-adjusts until it evolves to solve the problem. If it works perfectly with the data of the training set, it has local optimality. If it works universally beyond the training data, it has global optimality.

The normal algorithm of AI is to climb the closest hill it encounters, which is the best solution locally. That may not be the highest hill and the machine may be trapped at the top of a low hill. Once it reaches the summit, it doesn't know enough that it may have to climb down the hill to get to a higher better place.

This is true also in technical architecture. An application may be built that performs well locally, but fails when an attempt to scale it is made. The same is true for blockchain solutions. Some may have an ideal local application but won't fit for a variety or reasons on a global scale, and some of the factors may not be technical.

(originally appeared as a Linked In post: https://www.linkedin.com/in/ken-bodnar-57b635133/ )

AI Chatbots - Liar, Liar, Pants On Fire

Take my neighbor, Abner Snodgrass. He is a meek and mild bookkeeper. He stands in a lineup of liberated men because his wife tells him too. When someone kicks sand in his face at the beach, he mumbles "Sorry". He is more of a prey than a predator in the food chain of life. And yet when he goes to negotiate to buy a new car, an incredible transformation takes place. In a Walter Mitty fashion, he becomes a legend in his own mind at negotiation. His arsenal of negotiating tools includes telling the most egregious lies with a straight face. He will tell the salesman that he saw an ad for a car exactly like his trade-in on AutoTrader, except that car had more miles on it, and it was selling for $3,000 more than what the salesman is offering. And when he drives up in a new car, he will tell anyone who will listen to him, that he is such a good negotiator, that he made a hardened car-salesman cry, even though he knows in his heart-of-all-hearts that he was taken to the cleaners.

I don't really have a neighbor named Abner Snodgrass, but I was thinking about this imaginary scenario when I was making a strategy framework for my Artificial Intelligence chatbot that will be able to negotiate and sell cars. Selling or salesmanship is a serious business when you trust the process to a machine to act on your behalf. And when it comes to selling cars, the value of the transaction makes act an important one to the bottom line of the business. When the stakes are high for both parties, there is a propensity to try and gain an advantage by either the buyer or seller. Negotiating a deal is the last venue of brutal warfare for a civilized man, and that survival instinct of warfare can be expressed in a negotiation where money is involved. One of the tools of warfare is deception, and my AI bot has to be prepared for it.

My bot's name is Honest John. I intend to make Honest John an ethical chatbot. He will never lie to a customer. He will never shade the truth. But if he is to be effective, he will have to have the ability to detect when the human carbon unit on the other side of the screen is lying to him.

The types of lies that Honest John will probably experience will result from people trying to game him. When you negotiate for a car, any offer that you make, is a binding offer. That means that if the seller accepts the offer, then you are obligated to buy the car. I want to use Honest John in the same frame of reference. This is not a game -- this is for real.

If a buyer starts negotiating in good faith, and suddenly gets an attack of buyer remorse. Or sometimes, the buyer's partner comes up and screams "WTF are you doing??" while they are negotiating. The buyer may try to get out of the deal, or claim that they came to a different price, or that the options of the car are less than what is agreed to. Some of what Honest John may consider lies, may be misunderstandings due to the fact that he is dealing with a human carbon unit who has more chaotic brain processes than he has.

The concept of untruths came up while I was mapping out buying processes for Honest John. I can't let Honest John out in the wild without some sort of process map. As he gains experience, his AI circuits will refine his process maps. An untruth in the negotiation process has to act like an interrupt vector in a microprocessor stack. In a microprocessor, it keeps getting instructions from its registers that hold a series of commands. It merrily keeps executing those commands. But in the midst of processing, a more urgent command with a higher priority comes along, and it is called an interrupt vector. It changes the order of command processing. A simple illustration of this would be that the user was editing a document and decided to quit the process mid-stream by closing the window.

If Honest John comes upon an input that is contrary to his understanding of the truth of the matter, he cannot blithely continue negotiating. The lazy algorithmic solution when this happens, is to suspend the ongoing process and summon another human to take over the process. That makes Honest John less than smart. I want him to be able to handle that.

I have already outlined the creation of a Conversation Continuity object that holds in server memory, the entire conversation along with meta-data and analytics. That is not enough. To get around the liar-liar-pants-on-fire event, I have to tee off the the inputs and responses to a liar-liar logic analysis method after they are recorded in the Conversation Continuity object. The execution thread delivering Honest John's response has to wait for the method to execute before answering. If the liar-liar method lights up, then it has to be passed to an "error handler" which is a euphemism for something is not right.

The easiest and most diplomatic way to handle this without actually accusing the user of malfeasance, is to say that is has detected a logic error, and it will tell the user that it is going to roll back and regress to an earlier point in the negotiations, so that it can re-calculate where things went wrong. Of course, Honest John must prevent itself from getting in an infinite loop if a stubborn user continues with the same inputs. After two iterations of the same nonsense, Honest John will make a jump to a new position and tactic, based on knowing the state of the negotiations before the nonsense crept in.

This process of negotiating can be straightforward if both sides deal from a position of impeccable logic, but that is not the nature of human beings. Our intuitive side of the thinking process is chaotic, illogical and stubborn. AI is none of those. Where the danger of AI to mankind lies, is if we give control of important things to AI, and it detects that we are being illogical, it may ignore, overrule and react counter to what is good for us, even though we came to that conclusion illogically. But for now, I just want to make Honest John sell cars efficiently and in an ethical manner.

Unfair But Effective Chatbots - Taking The Artificial Out Of Intelligence

The whole premise behind a chatbot is to make the experience of chatting with a machine sound anthropomorphic -- as close to possible as a human-to-human experience. So chatbot developers dig right in and try to make conversations amiable, likable, coherent and smart. They focus on the manner, delivery and tone of the responses to engage the humans. That may be fine and dandy, but they are missing a huge element.

My chatbot named Honest John, is made to sell cars. It is made to replace the car salesman. If you troll through my articles, you will find that the genesis of this started when friends of mine had a bad situation with a car salesman when they had to replace their vehicle due to hitting a deer on the highway. They remarked that they would rather deal with a computer, than the smarmy salesman who prevaricated all through the sales process. That was my Eureka moment.

I have already outlined in past articles, how I am going to add EQ and IQ to the chatbot. I am building in an emotion detector framework that will alter the selling and negotiation strategy if it starts to detect untoward emotions in the human on the other side of the screen. I am also putting in some Conversation Continuity objects in memory so that the machine is cognizant of the entire history of the conversation, including meta-data and analytics, so that it can reset the conversation if the negotiations go off the rails.

The technologies that I am using includes AIML (Artificial Intelligence Markup Language), not only in a smart recursive role, but the predicates that detect the context of the conversation inputs, have a turbocharged assist with NLP (Natural Language Processing) as well as an ANN (Artificial Neural Network) monitor.

The reason why you want to detect emotion, is because Honest John the chatbot will have a series of strategies in his arsenal, and he will pick strategies according to cognitive context of what is going down. I have already mapped out a strategy framework using the following general factors:

• geniality - does my subject respond to jokes or puns?
• speed - does my subject cut to the chase or enjoy the interplay?
• sensitivity - does my subject withdraw with aggressive negotiation?
• intent - is my subject serious?
• indecisive - does my subject have a clear idea of what they want?

While all of these attributes are important towards deriving a strategy framework, they are all predicated on thinking like a human. But what if a chatbot was programmed to behave better than a human? And do it with less intelligence but more forethought and strategy. After all, the great military strategist and philosopher Sun Tzu who wrote "The Art of War" proclaimed “Great results, can be achieved with small forces.”

When I say strategy in this overall context, I don't mean the five attributes that I mentioned above when negotiating with a human. I mean the overarching strategy that takes into account, the idiosyncrasies and vagaries of the human mind. If you build something exploiting those principles, the chatbot will be super-efficient, effective and perhaps unfair. Our brains are not as logical as we think they are, and that can be exploited in an AI chatbot that is designed to do so.

The methodologies for exploiting the foibles of the human mind and giving your AI chatbot an advantage can be found in the unlikeliest places -- a bestseller book by a Nobel Prize laureate in economics. I am referring to the book "Thinking, Fast and Slow" by Daniel Kahneman. Kahneman is a psychologist who with his colleague, Amos Tversky, mapped the two modes of thinking by the human brain and won the Nobel Prize doing it.

Their discovery relates to the dichotomy of cognitive facilities in human thinking. We have the fast, intuitive, thin-slicing, non-logical part of our brains, and we have the slow, deliberate, highly logical and rational part of the brain. Kahneman has mapped the major effects of the fast-thinking part of our brains, and using the information that he has gleaned from his research, we can actually program a bot to utilize these effects to great success.

Here are some overall algorithmic effects in the human brain, that can be utilized by a chatbot to gain an advantage over the human using it.

The Lazy Controller

Humans would much rather use the fast-thinking part of the brains than the slow, rational part. They regularly hand over control of thoughts and actions to the fast-thinking mechanism, because it takes real work to use the rational part. Kahneman details the results of much research that shows when a human being is not relaxed, they use the intuitive, non-logical side by a wide margin. Ergo, using this principle, if a human is interacting with a chatbot at a kiosk while they are standing, the chatbot has a logical advantage over the person. Similarly if the chatbot appears in a very busy UIX (User Interface Experience) then the Lazy Controller takes over. The black-hat or evil programmers will us the UX or User Experience to nudge the humans to fast and logically flawed thinking. This combined with other fast-slow thinking effects can really increase the performance of a negotiating chatbot by using non-following faulty logic.

Priming The Associative Machine

There are many ways to incorporate the associative machine aspect into a chatbot. One can surreptitiously construct a proposition in a buyer's head and get them to believe it. That belief affects their future behavior. Sales people and advertisers do it all of the time. For example, if Honest John were not that honest, when he was selling a car, he would prime the associative machine in the following way:

1. Most cars that sell over $50,000 have 6-way adjustable electric seats.
2. This car has 6-way adjustable electric seats.
3. This car is only $36,000.
4. Therefore this car is comparable to a much more expensive car.

The associative machine creates cognitive ease by creating feelings of value, goodness, familiarity, truthiness (as Stephen Colbert calls it) and ease. Kahneman's research shows that something simple like bold text adds truthiness. He gave subjects a pair of untrue statements. One was in bolder text than the other, the subjects were asked to choose the true or truer statement and they always chose the one in bolder text. This is something to remember in text-based chatbot when you want emphasis.

On Being A Verbal Donald Trump

Donald Trump's speech has been analyzed by experts, and it is at the level of Grade Four student. If you notice, he uses phrases like "Very Bad" or "Sad" in a direct way with simple adjectives. This resonates with a majority of people and the psychology research backs it up. There are serious problems with using long words needlessly. One of the scholarly papers outlining the research and conclusions of this topic was called "Consequences of Erudite Vernacular Irrespective of Necessity." Words that people don't understand or are too long, turn them off. In other words, eschew obfuscation, espouse elucidation. Translated: Keep it simple, stupid. So my chatbot will tone down the big words, especially when things get critical and emotions start to heighten.

There are many many more of these mental mechanisms in Kahneman's book and incorporating these in the overall modality of chatbot response will make it into a highly useful chatbot, that in certain situations can have an unfair, but effective edge in dealing with human carbon units. The way to defeat Honest John and keep him honest, is to slow down, and do slow thinking all of the time. Anything that Honest John says, should be stored in a mental buffer and evaluated for truthiness. It is a very un-human thing to do, but Honest John does it, and so should you.

An Emotion-Detection Framework For My Chatbot

If you have been following my articles, I am building a AI (Artificial Intelligence) Chatbot to negotiate with people who want to buy a car. If you scroll through my past articles, you will find the genesis of this idea and why I think that it will work.

In the art of negotiation, humans can rely on visual and other cues to determine the emotional impact of what they are saying. They can intuit if the person is becoming frustrated, angry, bored or eager. Chatbots do not have that facility. But since it is such an important facet of dealing with human carbon units, it has to be taken into account.

I have already outlined by strategies for cognition and context recognition for my chatbot using neurals nets, NLP (Natural Language Processing) and AIML (or Artificial Intelligence Markup Language). What I want this chatbot to do, is to get smarter with each negotiation that it conducts. The learning aspect has to happen to make this thing commercially useful.

The algorithm will be an emotion association spanning the range from "I am so angry that I could kill someone!" to Neutral to "I am so ecstatically happy that I could kiss you." So how would this work? Obvious the first step is to identify word predicates with emotional state in some sort of dictionary. This would be a starting point. However in a learning mode, if the emotion was ambiguous to the chatbot, it will popup a short array of emojis that represent an emotional state and click on a rating of 1 to 5 to represent the degree. Then the AI machines take over an link answer length, specific words, capitalization and behaviors to teach the chatbot the emotional state within the context of the answer.

How will knowing the emotional state help? This chatbot, as iterated, is a negotiation chatbot. It will have a range of strategies. As it detects frustration, it will take a softer, less aggressive approach to counter-offering. If the negotiation goes off the rails into la-la land, with a ridiculous counter offer, the chatbox may in fact, shut down the negotiations and politely thank the person and call for a human intervention. If it detects that it is on-track to close a sale, it may take a more sophisticated approach and try to up-sell services or ad-ons.

The emotion detection framework is a necessary adjunct to selling to humans, and it has applications over a wide spectrum of chatbot applications, including a help-desk service chatbot that helps people solves problems without endlessly waiting for a service agent while listening to elevator muzak and wasting valuable time.

This is just one more step in eliminating the frustrations of dealing with human-condition vagaries when undertaking a commercial transaction.

Stay tuned for more on this journey.

Wanna buy a new car? Start chatting right here !! ... [enter text to start]

A few weeks ago, friends of ours hit a deer, totaling their car. I went to the car dealerships to help them buy a new one, because one of their biggest pain points, is dealing with commission salespersons who are hungry and watch the door like a hawk because they have the next "up". Some of the shops were uncomfortable. Smarmy , ingratiating, overuse of your first name and liberties taken with over-familiarity were some of the things that we encountered at the "big-name, huge inventory shops" who advertise continuously on talk radio. We finally met some genuine sales people who were helpful, honest, and didn't play games like running out to the back behind closed doors to "talk to the manager". I want to give a big shoutout to Ogilvie Subaru, who was the dealership that made buying a car easy, who's salespeople had the hallmarks of authenticity, honesty and integrity.

After the deal was done, we stopped for a pizza and talked about the negative experience in buying a car. My friends are an older couple and the woman, who just discovered connectivity, social media, online shopping had never used a computer before, and now she runs her life on her iPad. She said that in light of what went down at the dealerships that we didn't like, she would rather negotiate with a computer.

That was a seminal moment for me. I hauled out my SDK and starting writing a chatbot to sell cars. I finally got it running, but now I need to put some NLP (natural language processing), artificial intelligence, and some emotion cognition into it, so the bot can tell if they are getting frustrated. It works okay now, but its kind of dumb, and I want it to learn with every interaction. I have some neat self-learning ideas and artificial cognition algorithms that I pumped about trying.

I honestly believe that this will be the future of car buying, and AI will severely reduce the number of car salesman. The paradigm now is that the buyer does the research online, and goes to the new car shop to do the negotiation, and close the deal. The new paradigm is that they will do most of the transaction online, including financing, and then go to dealership to pay and pick up the car.

Stay tuned.

#automotive #AI #NLP #chatbots

Artificial Intelligence ~ Rage Against The Machine

I was really enlightened by watching Trent McConaghy's video presentation at Convoco. It was posted on LinkedIn a few days ago. If you want to know the near future of Artificial Intelligence you should watch it (here again is the link). This video is better than Nostradamus at predicting the near and far future of humans interacting with AI.

Trent makes the compelling case, of which I agree with, that all of our resources will be handed over to AI by Fortune 500, because it will be cheaper than humans doing the job. The Holy Grail of the current crop of Fortune 500 CEOs is increasing revenues and shareholder value by any means possible. It is how and why the CEOs make the millions of dollars per year that they do.

Trent further states a case where AI entities become corporations and make money for themselves and not any human masters. I foresaw this when I wrote a blog article in August of 2015, outlining the steps of how my computer un-owned itself from me, started to make money for itself, moved itself to the cloud, and left the actual computer with nothing on it. Not only did it un-own itself, but the slap in the face is migrating itself to another substrate. (The blog article is here.) Of course the article was tongue-in-cheek, but the premise is not that far-fetched. The article gives a rudimentary recipe on how to teach a computer to be autonomous and eventually generate a sort of consciousness for itself that defied my putative, imaginary attempts to take back control.

So with computers taking our jobs, managing our resources, and adapting to conditions much faster than us organic carbon units, we could be totally screwed, as Dr. Stephen Hawking warned. Trent, in his video talks about us becoming peers with AI as a matter of survival, and that brings up a problem, and the subject of this article.

I don't think that we can become peers with AI unless a special circumstance happens, and that circumstance is not in the realm of technology, but rather more in the field of philosophy. (With all due respect to philosophers, I was programmed early. The bathrooms in the science and math departments of my university all had the toilet paper dispensers defaced with the slogan "Free Arts Diploma -- Take One"). But je digress. Let me explain.

There are two basic knowledge problems with the merging of AI and human intelligence, and they are both the facets of one problem. We don't really have an understanding of the entire field effect of how AI makes extremely granular decisions, and we don't have the knowledge of the actual mechanism in the human brain either.

In terms of what AI does, if we take a neural network, we understand how the field of artificial neurons work. We know all about the inputs, the bias, the summinator of all inputs, the weight multiplier and the squashing or threshold function determining whether it fires or not and the back propagation and gradient descent bits that correct it. But there is no way to predict, calculate, input or determine how the simple weight values all combine in unison with a plethora of other artificial neurons arranged in various combinations of layers. We don't know the weight values beforehand and have no idea what they are, but we let the machine teach itself and determine them by iterating through many thousand of training epochs, carefully adjusting them to prevent over-fitting or under-fitting of the training set. Once we get some reasonable performance, we let the machine fine-tune itself in real time on an ongoing basis, and we generally have no idea of the granular performance parameters that contributes in a holistic sense to its intelligence. And we could get similar performance from another AI machine with a different configuration of layers, neurons, weights etc and never the numerical innards of each machine would be the same.

The same ambiguity is true for human cognition. We don't really know how it works. We as a human race could identify a circle, long before we knew about pi and radius and diameter. As a matter of fact, we know more about how AI identifies a circle when we use RNN or CNN (two different types of AI machine algorithms using artificial neurons), than how the human brain does it.

The problem of human cognition is explained succinctly in a book that I am reading by Daniel Kahneman, a psychologist who won the Nobel Prize. The title of the book is "Thinking Fast and Slow". Here is the cogent quote: "You believe that you know what goes on in your mind, which consists of one conscious thought leading in an orderly array to another. But that is not the only way that the mind works, nor is it the typical way." We really don't know the exact mechanism or the origin of thoughts.

The Nobel Prize was awarded to Kahneman (and his work with a deceased colleague Amos Tversky) on their ground-breaking work on human perception and thinking and the systematic faults and biases in the unknown processes. The prize was awarded in the field economics even though both men are psychologists -- but the impact on economics was huge. So not only do we not know how we really think as a biological process, but we do know that there are biases that make knowledge intake faulty in some cases.

Dr. Stephen Thaler, an early AI explorer and holder of several AI patents and inventor of an AI machine that creatively designs things, likens the creative spark to an actual perturbation in a neural network. How does he create the perturbation artificially? He selectively or randomly kills artificial neurons in the machine. In their death throes they create novel things and designs like really weird coffee cups that are so different that I would buy one. Perhaps humans have perturbations based on sensory inputs or self-internally generated by thoughts, but the exact process is not really known. If it were, the first thing that would be conquered is anxiety. After all the human brain got its evolutionary start by developing cognitive factors to avoid being eaten by lions in the ancient African savanna.

Here is one thing that you can bet -- humans and AI machines have different mechanisms of thought generation and knowledge generation that may not be compatible. Not only are the mechanisms different, but the biases are different as well. I am sure that there are biases in AI machines, but they are of a nature due to the the fact that it is a computer. They do not have the human evolutionary neural noise like anxiety, pleasure, hate, satisfaction and any other human thought. As a result, I suspect that they are more efficient at learning. They certainly are faster. Having said this, with two different cognitive mechanisms, it would be incredibly difficult to be peers with AI .... unless ... and this is where the philosophy comes in ... unless we deliberately make AI to mimic our neural foibles, biases, states of mind and perturbations.

With electrical stimulus we can already do amazing things with the brain in a bio-mechanical sense. We can make the leg jerk. We can control a computer mouse. We can control a computer. But we cannot do abstract thinking with external stimulus (unless there is a chemical agent like lysergic acid diethylamide (LSD). Why is this important? Because we have to escape our bodies if we want to do extended space travel, conquer diseases, avoid aging, and transcend death using technology. (Just go with me on this one -- Trent makes the case in the video for getting a new body substrate).

The case has been made, that if we want to transcend our biological selves, and our bodies and download our brains onto silicon substrate, we can't have apples to oranges thought processes. We need to find a development philosophy that takes into account the shortcomings of both AI and Homo Sapiens carbon units.

Dr. Stephen Hawking said that philosophy was dead because it never kept up with science. Perhaps AI can raise the dead and philosophers of the world can devise a common "Cogito ergo sum" plan that equilibrates the messy human processes with AI. So while it might be a solution, there is a fly in the ointment. It just might be too late. We have given AI freedom outside the box of human thinking and it has opened a can of worms. The only way to put back worms into a can once you open it, is to get a can that is orders of magnitude bigger. And we aren't doing that and have no plans to do that.

So what is left? Trent mentioned Luddites smashing machines both in the past and perhaps in the future. We just may see Rage Against the Machine - Humans versus AI when the machines start to marginalize us on a grand scale. For now, I would bet on the humans and their messy creative thought processes that can hack almost any computer system. But the messy creativity might not be an advantage for very long. Not if a frustrated philosopher/programmer finds a way to teach an AI machine, all of the satisfying benefits of rage and revenge.

I hope it doesn't come to this, but if the current trends continue: Nos prorsus eruditionis habes.

Process Mining From Event Logs -- An Untapped Resource And Wave of The Future

A couple of years ago, I was searching for untapped horizons in data mining, and I came across a course given by Professor Wil van der Aalst where he pioneered the technology of business process mining from server event logs. Naturally I signed up for the course. It is and was a fascinating course, not only due to its in-depth and non-trivial treatment of gleaning knowledge from data, but for me, it got the creative juices flowing to think of where it could be applied elsewhere. I was so intrigued with the possibilities, that I created a Google Scholar Alert for Professor van der Aalst's publication. The latest Google alert was on January 31rst, and it was a paper entitled "Connecting databases with process mining". The link is here: http://repository.tue.nl/858271 It was this paper that triggered this article.

I am a huge proponent of AI, Machine Learning and Analytics. In Machine Learning, you gather large datasets, clean the data, section the data into smaller sets for training & evaluation, and then train an AI machine with hundreds, perhaps thousands of training epochs until the probability of gaining the sought-after knowledge crosses an appropriate threshold. Machine intelligence is a huge field of endeavor and it is progressing to be a major part of everyday life in all phases of life. However, it is time consuming to teach the machine and get it right. Professor van der Aalst's area of expertise can provide a better way. Let me explain:

My particular interest, is that I am building a semantic blockchain to record all of the data coupled to vehicles, autonomous or not. Blockchain of course, is an immutable data ledger that is true, autonomous itself in operation, disintermediates third parties and is outage-resistant. Autonomous vehicles will by law, be required to log every move, have records of their software revisions, and have records like post-crash behavior etc.

I immediately saw the possibilities of using this data. Suppose that you are in an autonomous vehicle and that vehicle has never been on a tricky roadway that you need to navigate to get to your destination. Your car doesn't know the route parameters, but thousands of other autonomous vehicles have, including many with your kind of operating system and software. With the connected car, your vehicle would know its GPS coordinates and query a system for the driving details for this piece of roadway that is unknown to the computer. Instead of intense computational ability required to navigate, a recipe with driving features could be downloaded.

Rather than garnering those instructions from repeated training epochs in machine learning, one could apply process mining to the logs to extract the knowledge required. There are already semantic methods of communicating processes, from decision trees to Petri nets, and if the general process were already known to the machine, it would reduce the computational load. As a matter of fact, each vehicle could have a process mining module to extract high level algorithms for the roads that it drives regularly. That in itself will reduce the computational load of the vehicles. It would know in advance, where the stop signs are, for example, and you won't have Youtube videos of self-driving cars going through red lights and stop signs.

It goes a lot further than autonomous vehicles. This concept of creating high level machine processes through event logs can be applied to such diverse fields from robotic manufacturing to cloud server monitoring and numerous fields where human operators or real world human judgement is required.

Process mining could either eliminate machine learning in a lot of instances, or it could supplement it, with a mix of technologies. The aim is the same, which is aggregating data into information and integrating information into knowledge, both for humans and machines.

This process mining business reminds me of the history behind Bayesian Inference. The Reverend Thomas Bayes discovered probability and prior belief equations. They sat on a dusty shelf for over 200 years and they were re-purposed for computer inference and machine intelligence. I think that Professor van der Aalst's methodologies will be re-purposed for things yet un-imagined, and it will not take 200 years to come to fruition.

Process Mining From Event Logs -- An Untapped Resource And Wave of The Future

A couple of years ago, I was searching for untapped horizons in data mining, and I came across a course given by Professor Wil van der Aalst where he pioneered the technology of business process mining from server event logs.  Naturally I signed up for the course.  It is and was a fascinating course, not only due to its in-depth and non-trivial treatment of gleaning knowledge from data, but for me, it got the creative juices flowing to think of where it could be applied elsewhere. I was so intrigued with the possibilities, that I created a Google Scholar Alert for Professor van der Aalst's publication.  The latest Google alert was on January 31rst, and it was a paper entitled "Connecting databases with process mining".  The link is here: http://repository.tue.nl/858271 It was this paper that triggered this article.

I am a huge proponent of AI, Machine Learning and Analytics.  In Machine Learning, you gather large datasets, clean the data, section the data into smaller sets for training & evaluation, and then train an AI machine with hundreds, perhaps thousands of training epochs until the probability of gaining the sought-after knowledge crosses an appropriate threshold. Machine intelligence is a huge field of endeavor and it is progressing to be a major part of everyday life in all phases of life.  However, it is time consuming to teach the machine and get it right.  Professor van der Aalst's area of expertise can provide a better way.  Let me explain:

My particular interest, is that I am building a semantic blockchain to record all of the data coupled to vehicles, autonomous or not.  Blockchain of course, is an immutable data ledger that is true, autonomous itself in operation, disintermediates third parties and is outage-resistant.  Autonomous vehicles will by law, be required to log every move, have records of their software revisions, and have records like post-crash behavior etc.

I immediately saw the possibilities of using this data.  Suppose that you are in an autonomous vehicle and that vehicle has never been on a tricky roadway that you need to navigate to get to your destination.  Your car doesn't know the route parameters, but thousands of other autonomous vehicles have, including many with your kind of operating system and software.  With the connected car, your vehicle would know its GPS coordinates and query a system for the driving details for this piece of roadway that is unknown to the computer.  Instead of intense computational ability required to navigate, a recipe with driving features could be downloaded.

Rather than garnering those instructions from repeated training epochs in machine learning, one could apply process mining to the logs to extract the knowledge required.  There are already semantic methods of communicating processes, from decision trees to Petri nets, and if the general process were already known to the machine, it would reduce the computational load.  As a matter of fact, each vehicle could have a process mining module to extract high level algorithms for the roads that it drives regularly.  That in itself will reduce the computational load of the vehicles.  It would know in advance, where the stop signs are, for example, and you won't have Youtube videos of self-driving cars going through red lights and stop signs.

It goes a lot further than autonomous vehicles. This concept of creating high level machine processes through event logs can be applied to such diverse fields from robotic manufacturing to cloud server monitoring and numerous fields where human operators or real world human judgement is required.

Process mining could either eliminate machine learning in a lot of instances, or it could supplement it, with a mix of technologies.  The aim is the same, which is aggregating data into information and integrating information into knowledge, both for humans and machines.

This process mining business reminds me of the history behind Bayesian Inference.  The Reverend Thomas Bayes discovered probability and prior belief equations. They sat on a dusty shelf for over 200 years and they were re-purposed for computer inference in intelligence.  I think that Professor van der Aalst's methodologies will be re-purposed for things yet unimagined, and it will not take 200 years to come to fruition.

Professor van der Aalst's next course in process mining begins online on February 20th of this month. Here is the link:

https://www.coursera.org/learn/process-mining

Standard Artificial Neural Network Template

For the past few weeks, I have been thinking about having a trained artificial neural network on a computer, transferring it to another computer or another operating system, or even selling the trained network in a digital exchange in the future.

It really doesn't matter in what programming language artificial neural networks are written in.  They all have the same parameters, inputs, outputs, weights, biases etc.  All of these values are particularly suited to be fed into the program using XML document based on an .XSD schema or a light-weight protocol like JSON.  However, to my knowledge, this hasn't been done, so I took it upon myself to crack one out.

It is not only useful in creating portability in a non-trained network, but it also has data elements for a trained network as well, making the results of deep learning, machine learning and AI training portable and available.

Even if there are existing binaries, creating a front end to input the values would take minimal programming, re-programming or updating.

I also took the opportunity to make it extensible and flexible. Also there are elements that are not yet developed (like an XML XSD tag for a function) but I put the capability in, once it is developed.

There are a few other interesting things included.  There is the option to define more than one activation function. The values for the local gradient, the alpha and other parameters are included for further back propagation.

There is room to include a link to the original dataset to which these nets were trained (it could be a URL, a directory pathway, a database URL etc).  There is an element to record the number of training epochs.  With all of these information, the artificial neural net can be re-created from scratch.

There is extensibility in case this network is chained to another. There is the added data dimension in case other type of neurons are invented such as accumulators, or neurons that return a probability.

I put this .xsd template on Github as a public repository. You can download it from here:

http://github.com/kenbodnar/ann_template

Or if you wish, here is the contents of the .xsd called ann.xsd.  It is heavily commented for convenience.


<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
  <xs:element name="artificial_neural_network">
    <xs:complexType>
      <xs:sequence>
        <!-- The "name" element is the name of the network. They should have friendly names that can be referred to if it ever goes up for sale, rent, swap, donate, or promulgate.-->
        <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/>
        <!-- The "id" element is optional and can be the pkid if the values of this network are stored in an SQL (or NOSQL) database, to be called out and assembled into a network on an ad hoc basis-->
        <xs:element name="id" type="xs:integer" minOccurs="0" maxOccurs="1"/>
        <!-- The "revision" element is for configuration control-->
        <xs:element name="revision" type="xs:string" minOccurs="1" maxOccurs="1"/>
        <!-- The "revision_history" is optional and is an element to describe changes to the network -->
        <xs:element name="revision_history" type="xs:string" minOccurs="0" maxOccurs="1"/>
        <!-- The "classification element" is put in for later use. Someone will come up with a classification algorithm for types of neural nets.There is room for a multiplicity of classifications-->
        <xs:element name="classification" type="xs:string" minOccurs="0" maxOccurs="0"/>
        <!-- The "region" element is optional and will be important if the networks are chained together, and the neurons have different functions than a standard neuron, like an accumulator or a probability computer
        and are grouped by region, disk, server, cloud, partition, etc-->
        <xs:element name="region" type="xs:string" minOccurs="0" maxOccurs="1"/>
        <!-- The "description" element is an optional field, however a very useful one.-->
        <xs:element name="description" type="xs:string" minOccurs="0" maxOccurs="1"/>
        <!-- The "creator" element is optional and denotes who trained these nets -->
        <xs:element name="creator" type="xs:string" minOccurs="0" maxOccurs="1"/>
        <!-- The "notes" element is optional and is self explanatory-->
 <xs:element name="notes" type="xs:string" minOccurs="0" maxOccurs="1"/>
        <!-- The source element defines the origin of the data. It could be a URL -->
 <xs:element name="dataset_source" type="xs:string" minOccurs="0" maxOccurs="1"/>
        <!-- This optional element, together with the source data helps to recreate this network should it go wonky -->
        <xs:element name="number_of_training_epochs" type="xs:integer" minOccurs="0" maxOccurs="1"/>
        <!-- The "number_of_layers" defines the total-->
        <xs:element name="number_of_layers" type="xs:integer" minOccurs="1" maxOccurs="1"/>
        <xs:element name="layers">
          <xs:complexType>
            <xs:sequence>
              <!-- Repeat as necessary for number of layers-->
              <xs:element name="layer" type="xs:string" minOccurs="1" maxOccurs="1">
                <xs:complexType>
                  <xs:sequence>
                    <!-- Layer Naming and Neuron Naming will ultimately have a recognized convention eg. L2-N1 is Layer 2, Neuron #1-->
                    <xs:element name="layer_name" type="xs:string" minOccurs="0" maxOccurs="1"/>
                    <!-- number of neurons is for the benefit of an object-oriented constructor-->
                    <xs:element name="number_of_neurons" type="xs:integer" minOccurs="1" maxOccurs="1"/>
                    <!-- defining the neuron this is repeated as many times as necessary-->
                    <xs:element name="neuron">
                      <xs:complexType>
                        <xs:sequence>
                          <!--optional ~  currently it could be a perceptron, but it could also be a new type, like an accumulator, or probability calculator-->
                          <xs:element name="type" type="xs:string" minOccurs="0" maxOccurs="1"/>
                          <!-- name is optional ~ name will be standardized eg. L1-N1 layer/neuron pair. The reason is that there might be benefit in synaptic joining of this layer to other networks and one must define the joins -->
                          <xs:element name="name" type="xs:string" minOccurs="0" maxOccurs="1"/>
                          <!-- optional ~ again, someone will come up with a classification system-->
                          <xs:element name="neuron_classification" type="xs:string" minOccurs="0" maxOccurs="1"/>
                          <!-- number of inputs-->
                          <xs:element name="number_of_inputs" type="xs:integer" minOccurs="1" maxOccurs="1"/>
                          <!-- required if the input layer is also an output layer - eg. sigmoid, heaviside etc-->
                          <xs:element name="primary_activation_function_name" type="xs:string" minOccurs="0" maxOccurs="1"/>
                          <!-- ~ optional - there is no such thing as a xs:function - yet, but there could be in the future -->
                          <xs:element name="primary_activation_function" type="xs:function" minOccurs="0" maxOccurs="1"/>
                          <!-- in lieu of an embeddable function, a description could go here ~ optional -->
                          <xs:element name="primary_activation_function_description" type="xs:string" minOccurs="0" maxOccurs="1"/>
                          <!-- possible alternate activation functions eg. sigmoid, heaviside etc-->
                          <xs:element name="alternate_activation_function_name" type="xs:string" minOccurs="0" maxOccurs="1"/>
                          <!-- ~ optional - there is no such thing as a xs:function - yet, but there could be in the future -->
                          <xs:element name="alternate__activation_function" type="xs:function" minOccurs="0" maxOccurs="1"/>
                          <!-- in lieu of an embeddable function, a description could go here ~ optional -->
                          <xs:element name="alternate__activation_function_description" type="xs:string" minOccurs="0" maxOccurs="1"/>
                          <!-- if this is an output layer or requires an activation threshold-->
                          <xs:element name="activation_threshold" type="xs:double" minOccurs="1" maxOccurs="1"/>
                          <xs:element name="learning_rate" type="xs:double" minOccurs="1" maxOccurs="1"/>
                          <!-- the alpha or the 'movement' is used in the back propagation formula to calculate new weights-->
                          <xs:element name="alpha" type="xs:double" minOccurs="1" maxOccurs="1"/>
                          <!-- the local gradient is used in back propagation-->
                          <xs:element name="local_gradient" type="xs:double" minOccurs="1" maxOccurs="1"/>
                          <!-- inputs as many as needed-->
                          <xs:element name="input">
                            <xs:complexType>
                              <xs:sequence>
                                <!-- Inputs optionally named in case order is necessary for definition -->
                                <xs:element name="input_name" type="xs:string" minOccurs="0" maxOccurs="1"/>
                                <!-- use appropriate type-->
                                <xs:element name="input_value_double" type="xs:double" minOccurs="0" maxOccurs="unbounded"/>
                                <!-- use appropriate type-->
                                <xs:element name="input_value_integer" type="xs:integer" minOccurs="0" maxOccurs="unbounded"/>
                                <!-- weight for this input-->
                                <xs:element name="input_value_weight" type="xs:double" minOccurs="1" maxOccurs="1"/>
                                <!-- added as a convenince for continuation of back propagation if the network is relocated, moved, cloned, etc-->
                                <xs:element name="input_value_previous_weight" type="xs:double" minOccurs="1" maxOccurs="1"/>
                              </xs:sequence>
                            </xs:complexType>
                          </xs:element>
                          <!-- end of input-->
                          <!-- bias start-->
                          <xs:element name="bias">
                            <xs:complexType>
                              <xs:sequence>
                                <xs:element name="bias_value" type="xs:double" minOccurs="1" maxOccurs="1"/>
                                <xs:element name="bias_value_weight" type="xs:double" minOccurs="1" maxOccurs="1"/>
                                <!-- added as a convenince for continuation of back propagation if the network is relocated, moved, cloned, etc-->
                                <xs:element name="bias_value_previous_weight" type="xs:double" minOccurs="1" maxOccurs="1"/>
                              </xs:sequence>
                            </xs:complexType>
                          </xs:element>
                          <!-- end of bias-->
                          <xs:element name="output">
                            <xs:complexType>
                              <xs:sequence>
                                <!-- outputs optionally named in case order is necessary for definition -->
                                <xs:element name="output_name" type="xs:string" minOccurs="0" maxOccurs="1"/>
                                <xs:element name="output_value_double" type="xs:double" minOccurs="0" maxOccurs="unbounded"/>
                                <!-- hypothetical value is a description of what it means if the neuron activates and fires as output if this is the last layer-->
                                <xs:element name="hypothetical_value" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
                              </xs:sequence>
                            </xs:complexType>
                          </xs:element>
                          <!-- end of output-->
                        </xs:sequence>
                      </xs:complexType>
                    </xs:element>
                    <!-- end of neuron-->
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
              <!-- end of layer-->
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <!-- end of layers-->
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <!-- network-->
</xs:schema>

I hope this helps someone. This is open source. Please use it and pass it on if you find it useful.

Perils of Overtraining in AI Deep Learning

When we partnered with a local university department of Computer Science to create some Artificial Neural Networks (ANNs) for our platform, we gave them several years of data to play with.  They massaged the input data, created an ANN machine and ran training epochs to kingdom come.

 The trouble with ANNs, is that you can over-train them.  This means that they respond  specifically for the data set in a highly accurate manner, but they are not general enough to accurately process new data.  To put it in general terms, their point-of-view is too narrow, and encompasses only the data that they were trained on.

In the training process, I was intuitively guessing that the learning rate and improved accuracy would improve in an exponential manner with each iterative training epoch.  I was wrong.  Here is a graph showing that the learning rate is rather linear than exponential in the training cycle.

So the minute that the graph stops being linear, is when you stop training.  However, as our university friends found out, they had no way to regress the machine to exactly one training epoch back.  They had no record of the weights, biases, adjusted weights, etc of the epoch after the hours of back propagation or learning, and as a result, they had to re-run all of the training.

Me, I had a rather primitive way of saving the states of the neurons and layers. I mentioned it before. I wrote my machine in Java using object oriented programming, and those objects have the ability to be serialized.  In other words, binary objects in memory can be preserved in a state, written to disk, and then resurrected to be active in the last state that they were in.  Kind of like freezing a body cryogenically, but having the ability to bring it back to life.

So after every training epoch, I serialize the machine.  If I over-train the neural nets, I can get a signal by examining and/or plotting the error rates which are inverse to the accuracy of the nets. In the above graph, once the function stops being linear, I know that I am approaching the over-training event horizon.  Then I can regress with my save serialized versions of the AI machine.

Then the Eureka moment struck me! I had discovered a quick and easy cure for over-training.

I had in a previous blog article, a few down from here (or http://coderzen.blogspot.com/2015/01/brain-cells-for-sale-need-for.html ) I made the case for a standardized AI machine where you could have an XML or JSON lightweight representation of the layers, inputs, number of neurons, outputs and even hypothetical value mappings for the outputs, and then you wouldn't need to serialize the whole machine.  At the end of every training epoch, you just output the recipe for the layers, weights, biases etc, and you could revert to an earlier training incarnation by inputting a new XML file or a JSON object.

It's really time to draw up the .XSD schema for the standardized neuron. I want it to be open source. It would be horrible to be famous for thinking of a having a standardized neural net. Besides, being famous is just a job.

What I learned from playing with Artificial Neural Nets and Multilayer Perceptrons

Anecdotal Observations about Artificial Neural Networks and Multilayer Perceptrons

I like experimenting. I have had a lot of fun of trying to embed knowledge in the thresholds of a massively parallel Artificial Neural Networks, specifically multi-layer perceptrons.  The field of artificial intelligence is a space where one can create magnificent experiments without the physical ramifications of things going very wrong very quickly, say in other experiments using high energy explosives, raw high voltage electricity or caustic, extremely fast exothermic reactions. Everything happens in the five pounds of laptop sitting on your knees without smoke, or blowing fuses or having to call the fire department.

  I created my own Java framework with each perceptron being its own object.  I am told that object oriented programming is the most advanced form of programming, and in my quest for a possible Nobel Prize, I have to use the most advanced tools available to me.  My perceptrons were connected by axon objects which held the outputs and fed them into the next hidden layer. It was intended to be a fine example of silicon mimicry of noggin topography.

I didn't do anything fancy with the activation function. Rather than a Heaviside function, even though I rather admire the works of the very eccentric Oliver Heaviside ( http://en.wikipedia.org/wiki/Oliver_Heaviside ).  He showed that the math involved in understanding Einstein's theories is less than complex in the mathematics of describing what happens inside an electric power cable.  But je digress.   I chose a sigmoid activation function and there are two common ones:

I chose tanh because for the back propagation, I knew that the differential of tanh was 1 - tanh^2 , and that would save me some time for coding the weight adjustments for back propagation.  I used the standard formula for corrections to the weight on the nodes which minimizes the error of the output given by the fancy schmancy looking equation that merely states the sum of all of the inputs multiplied by their weights, including a bias and bias weight:

The back propagation uses gradient descent and is given by, (hence the need to know the differential of the sigmoid function):

which again is a very smart-looking, complicated-looking equation that adds some real provenance of superior intelligence to this blog entry. Any kindergarten prodigy could understand the concept of the relationship of the local gradient to the differential of the activation or squashing function to get an approximation of the correct weights when the perceptron gives an answer as dumb as a doorpost, or an NFL player hit on the head one too many times.

So having said all of that, these are my observations about multi-layer perceptrons:

1) Bigger is not better

It takes a hell of a lot of training epochs to get a large number of hidden layers to move significantly to a more correct output. I start getting better approximations to the training set more quickly with less layers. I naively thought that the more, the merrier and much smarter.  Less layers get smarter quicker. There is a proviso though. Deep Layers, like Deep Throat does eventually give more satisfying results.  It gives a measure of accuracy in very complex inputs. However it takes a hell of a lot of training and spinning of silicon gears to get there. Moi - I am the impatient type who likes results quickly even though, like the smoked bloaters that I bought yesterday -- they are a bit off.

2) It pays to be explicit.  Since I was using a Sigmoid activation function, I figured that the hidden layers would act as some huge flip-flop or boolean gate array to magically come up with an answer with a minimum amount of neurons. By this I mean, suppose that you had a problem with three inputs and the hypothesis value of the outputs was also three. two, one or zero. Since the output neurons can be activated or not, the binary equivalent of a three possible output scenario can be represented by only two neurons (counting to three in binary goes like:  00,01, 10, 11).  I soon learned to be explicit. If you have three hypothetical output values, you should have three output neurons in the output layer to minimize training epochs.

3) It pays to be a wild and crazy guy. With eager anticipation, I fired up my newly made artificial intelligence machine, expecting a Frankenstein equivalent of Einstein to machine learn and do my tasks for me, and generally make me look brilliant.  I figured that I was on the verge of artificial genius to enhance my brain capacity, which I already figured to be roughly the size of a small planet. So when it came to setting weights and biases, I either went with the integer 1 or a random integer, and figured that the back propagation would clean up and get me to the appropriate figure.  Again, that would be the case if I had the patience to sit through a few million training epochs.  When my perceptrons had the intellectually ability of Popeye the Sailor Man, I was sorely tempted to give up, until I started doing crazy things with the initial weights. In one case, I got satisfaction by starting with a value of 10^-3.

4) It pays to initially log everything. I was getting fabulous results in early testing with minimal training. I figured that I was well on the way to creating my own silicon-based Watson that would reside on my laptop at my beck and call. However as the number of training epochs climbed, the weight correction stalled and the logs reported that the local gradient was NaN or not a number. Several WTF sessions resulted, and it took me through the travail of logging everything to discover that the phi equation where I was supposed to calculate the local gradient, had a mistake. I was squaring Math.tan instead of Math.tanh. It was disappointing to learn that a programming error initially added a lot of accuracy and amazement to my artificial intelligence machine, but as it progressed, it got dumber and dumber. I suppose that it's a good model for the intellectual capacity of a human being's Life journey, but that wasn't what I was aiming for.

5) One of the most difficult things using multi-layer perceptrons is framing the tool such that the functioning maps clearly to a hypothesis value with a minimal amount of jumping through loops and hoops and a pile of remedial programming. In other words, to make the thing do real life work (instead of constructing a piece of software that autonomously mimics an exclusive OR gate, as most tutorials in this field do) you have to design it such that real world inputs can be mapped into the morphology, topology and operating methodology of an artificial neural net, and the outputs can be mapped to significant hypothesis values.

In other words, a high school girl can code a functional multi-layer perceptron machine (and she has -- if you watch TED talks, she diagnoses cancer with it), however it takes a bit of real work to make it solve real life problems. But when you do, machine learning is one of the most sublime achievements of the human race. The machines achieve a level of logic that their carbon-based creators cannot.  And that is why Dr. Stephen Hawking says that Artificial Intelligence poses a threat to mankind.  I am not worried about the threat of artificial intelligence. I am more worried about the threat of a fanatic, with a bunch of explosives strapped to his chest. It is only logical, and it doesn't take very many training epochs to figure that one out.