If you’re a budding startup entrepreneur looking to give you business some buzz with investors, “big data” is so last year and “cyber” so 2014. “Mobile” is not a lot better than “dot.com.” Nowadays any startup worth its salt is putting the words “deep learning” into its mission statement.
- Microsoft pulls robot after it tweets 'Hitler was right I hate the jews'
- The birth of a unicorn: The story of an Israeli startup worth a billion dollars
- 1923: A scientist who would quail at the artificial intelligence he created is born
Fortunately, deep learning is for real and better yet can apply to almost any technology because it involves giving computers the ability to imitate the processes that enable the human brain to recognize speech, decipher images and analyze texts. You can already find elements of deep learning in smartphones, apps, computers, cars and any device connected to the Internet.
Scientists and researchers have been trying for decades to develop computers or machines that can copy the complex processes that occur in the brain, chiefly the ability of people to learn. But real breakthroughs have only been achieved in the last two to three years that promise to bring the technology closer to the Holy Grail of true artificial intelligence.
Two weeks ago these improved capabilities were amply demonstrated with AlphaGo, software developed by Google’s DeepMind division that beat human adversary Lee Sedol 4-1 at the Chinese board game Go. The ability of a computer to succeed at chess is usually thought of as the gold standard for AI, but Go is actually much more complicated and last month’s victory marked a milestone in the age-old competition between man and machine.
A computational model using algorithms and infrastructure of neuronal networks that imitate the process of learning that occurs in the human brain, deep learning has not only captured the imagination of computer scientists but is set to become an integral part of the global high-tech industry. The U.S. consulting firm Deloitte estimates that 80 of the world’s 100 largest software companies will adopt software systems based on AI and cognitive technologies this year, a 25% increase over 2015. It predicts the AI market will grow to $43 billion by 2024.
Yet for all its sophistication, many of its applications are often for humans, at least, ordinary day-to-day activities, such as enabling a smartphone to understand human speech and respond intelligently and cars to drive themselves.
Learning is hard
“Learning is something humans do all the time unconsciously. We use information we receive to construct knowledge,” explains Prof. Shai Shalev-Schwartz, vice president of technology at Mobileye, an Israeli company that makes collision-prevention technology, and an expert on deep learning at Hebrew University. “If we show someone a picture of a car they know it’s a car, but they can’t explain how they know this. This is what I as a researcher am trying to discover and this is what deep learning systems do – try to understand rules. These are automatic tools for making inductions from the individual to the general.”
Deep learning, which is part a broader discipline known as machine learning, which in turn is part of the larger area of artificial intelligence, mimics the brain by simultaneously processing large amounts of data in a network of artificial neurons and synapses. The system teaches itself by identifying repeating patterns in the huge amounts of data fed into it.
In industry, deep learning enables computers to make sense of the vast quantities of data they can absorb and store. One application is computerized vision, which enables a computer to process images in real time, for instance monitoring the road for others cars and pedestrians while driving a car. Another is being able to understand speech and respond to it. Deep learning also can provide solutions for more complex tasks in areas such as medicine, computer security and finance.
Shalev-Schwartz says that Mobileye identified the critical role deep learning could play early on and many of the company’s systems are based on the technology. “In computerized vision, in which many aspects of a particular image are collected, it’s possible, for example, to convert the concept of a car into a rule. The real reason we’re pursuing this avenue is that we’ve been unsuccessful in constructing the entire process that will imitate what the brain does,” he says.
At Mobileye, computation is done using multiple levels of artificial neurons. Each layer processes the information it gets at a higher level than the preceding one and gradually a complex computation is made.
“We begin with a layer that looks at the pixels in a picture. The system can distinguish between lines and colors and then converts the information into more abstract concepts, such as identifying a window, a road or a license plate, for example. It then goes to a higher level. This is a model built on a series of steps at the end of which the system identifies the entire entity as a car,” says Shalev-Schwartz. “In fact, the more one delves into layers the ability to identify different objects is greater, so that by the end of this process, the system will be able to distinguish between a dog, a man or a car.”
Competing with flat
Even though the basic concept by deep learning and its model of artificial neurons have been understood for years, researchers long preferred to focus on “flat” models of data analysis. A significant breakthrough deep learning came in 1989, when Yann LeCun built a network that could identify handwriting and could be used for reading checks.
But after that deep learning stalled until the mid-2000s. According to Shalev-Schwartz, the real impetus came in 2012, when Geoffrey Hinton, Alex Krizhevsky and Ilya Sutskever, three leading researchers from the University of Toronto, demonstrated a significant leap in the ability of a neural network to understand a picture. “The academic and business communities realized that deep learning is very powerful,” says Shalev-Schwartz.
The three researchers later found themselves heading a lab at Google and in December 2013 Prof. LeCun became the director of AI research at Facebook.
Another major reason for the renewed interest in artificial neural networks is the huge amounts of data now accessible through cloud computing. Systems are now equipped with massive processing capabilities, with learning achieved through graphic processing units that can process data simultaneously through different channels, much like the brain does. GPUs can yield results much faster than regular CPUs.
Eden Shochat, a partner in the venture capital firm Aleph, was involved in one of the biggest Israeli success stories in computerized vision, with the sale of the face-recognition startup face.com to Facebook in 2012. He says a revolution is underway, but to enjoy the fruits of the technology a lot of high-quality and specific data is required and isn’t always there. “For example, [the Israeli health maintenance organization] Kupat Holim Clalit possesses a lot of radiological data. If the government would release medical information this would improve our lives, but there is no one who is compiling all this data on behalf of the state,” he says.
Looking at the Israeli startup scene, Shochat says deep learning has become a buzzword but fewer companies are really employing it. One that is is The Fifth Dimension, whose technology analyzes large amounts of data drawn from different sources to identify security threats in real time. Another is
Deep Instinct, which works in cyber security. Both companies are headed by Guy Caspi, formerly the CEO of Tamares. According to Israel Venture Capital Research, Deep Instinct has raised $35 million, mainly from the U.S. firm Blumberg Capital.
Eli David, a lecturer at the Computer Science Department at Bar-Ilan University and Deep Instinct’s vice president, says its software is able to detect 20% more malware than existing antivirus programs. “There are one million malicious files created daily and the number is growing. We’re interested in finding the new ones,” he explains.
By using deep learning algorithms, explains David, company teams are training the a system of artificial neural networks that consist of a few GPU processors, to identify which files are malicious and which ones aren’t. “We use tens of millions of malicious and non-malicious files. The artificial brain learns the characteristics of these files in a process lasting 48 hours. After that the analysis is instantaneous,” he says.
Deep Instinct, like Fifth Dimension, doesn’t reveals its customers’ names but hints they include a large telecoms company and several financial institutions.
New generation of smart apps
Big companies, however, are leading research and application in deep learning. Along with Google, which showed impressive abilities with DeepMind’s victory in the Go game, Microsoft, China’s Baidu, IBM and Facebook are investing heavily in this area. In contrast to the usual practice in a competitive market, in the world of AI, companies shares their developments and use open source code.
Shalev-Schwartz admits the Holy Grail has not been attained yet. “Models are being constructed to solve complex problems, but they often fail at simpler problems. There’s no panacea,” he told a recent conference.
There are also legitimate concerns about the increasing power of computers and their potential to cause real problems. “It would take one computer planning an attack on other ones to make the danger real,” Prof. Lior Wolf, an expert in the field at Tel Aviv University, told a cyber conference last week.
Shalev-Schwartz understands the concern but sounds more optimistic. “There are good and bad aspects and the question is how we scientists direct the research so that it remains safe.” But he says the day of danger remains far off. Quoting Prof. Andrew Ng, a leader in the field at Baidu: “Talking about this now is like talking about overpopulation on Mars.”