20/12/2016 10:52 AM IST | Updated 20/12/2016 12:52 PM IST

Meet Jarvis, The Personal Assistant Built By Mark Zuckerberg

Jarvis controls the Facebook CEO's home much like the one in the movie 'Iron Man'.

AFP/Getty Images

Facebook CEO Mark Zuckerberg loves to code or, in other words, write computer programs. Up until a couple of years after 2004, the year he founded Facebook, he continued to do a lot of coding for the company. He has since continued to code on small projects and, in 2012, he had set a challenge for himself to code every day. His 2016 challenge for himself involved making Jarvis, a personal assistant for his home that could be controlled by phone.

Here is what Zuckerberg had posted about the challenge online,

"My personal challenge for 2016 is to build a simple AI to run my home and help me with my work. You can think of it kind of like Jarvis in Iron Man.

I'm going to start by exploring what technology is already out there. Then I'll start teaching it to understand my voice to control everything in our home -- music, lights, temperature and so on. I'll teach it to let friends in by looking at their faces when they ring the doorbell. I'll teach it to let me know if anything is going on in Max's room that I need to check on when I'm not with her. On the work side, it'll help me visualize data in VR to help me build better services and lead my organizations more effectively"

Now, as 2016 comes to a close, he has almost completed his challenge. Yesterday, Zuckerberg posted how he built Jarvis over the year.

He started the process by trying to figure out how many things in his home were connected to the Internet and how many would need a hardware modification. Some examples he gave included connecting the toaster and the food processor at his home to an Internet-enabled switch.

Before turning Jarvis into a fully mobile- and voice controlled assistant, Zuckerberg had to teach it the natural language. So he started to build the code through which he could control different things in his house with the computer.

"It started simple by looking for keywords, like 'bedroom', 'lights', and 'on' to determine I was telling it to turn the lights on in the bedroom. It quickly became clear that it needed to learn synonyms, like that 'family room' and 'living room' mean the same thing in our home. This meant building a way to teach it new words and concepts," Zuckerberg posted.


He explained that while Artificial Intelligence has evolved, it is still important to teach it context, such as identifying people present in a room, to check if his baby has woken up, play music based on the mood specified in the voice command and more.

Zuckerberg installed a bunch of cameras and made a central server to make Jarvis learn about different places in the house. He has even built a face recognition algorithm into Jarvis so that if someone is at the door, Jarvis can recognise her or him.

While Zuckerberg has used a lot of Internet of Things (IoT) technologies, he has also incorporated many Facebook tools such as the Facebook Messenger bot. While the bot had voice recognition built into it, Zuckerberg found that he preferred using text mostly as that was non-intrusive.

"One thing that surprised me about my communication with Jarvis is that when I have the choice of either speaking or texting, I text much more than I would have expected. This is for a number of reasons, but mostly it feels less disturbing to people around me," he said.

"Another interesting limitation of speech recognition systems -- and machine learning systems more generally -- is that they are more optimized for specific problems than most people realize. For example, understanding a person talking to a computer is the subtly different problem from understanding a person talking to another person," Zuckerberg said, speaking about speech recognition systems.


While this looks all fancy, Zuckerberg he says that there are still many improvements that need to be done. Many times it takes more than one attempt to switch on the lights or switch the TV on for him or his wife, Priscilla Chan. In an interview with the FastCompany magazine, he cited many instances of vulnerability in Jarvis.

"One of the rooms that is adjacent to the [TV] room is... Priscilla's office," Zuckerberg says in the interview. "So we had this funny thing for a while where... we're going to watch TV, and [Jarvis] would just turn off all the lights downstairs, and she'd be trying to work, and she'd be like, 'MARK!'"

The next step for Zuckerberg is to build a system on Android phones and keep them in the every corner of the house where they can keep listening to commands. A system, in other words, like Amazon Echo and Google Home. He said that Jarvis was not production ready yet but it would be in the coming year.