computers

AI is about to completely change how you use computers

Posted by

0


I still love software as much today as I did when Paul Allen and I started Microsoft. But—even though it has improved a lot in the decades since then—in many ways, software is still pretty dumb.

To do any task on a computer, you have to tell your device which app to use. You can use Microsoft Word and Google Docs to draft a business proposal, but they can’t help you send an email, share a selfie, analyze data, schedule a party, or buy movie tickets. And even the best sites have an incomplete understanding of your work, personal life, interests, and relationships and a limited ability to use this information to do things for you. That’s the kind of thing that is only possible today with another human being, like a close friend or personal assistant.

In the next five years, this will change completely. You won’t have to use different apps for different tasks. You’ll simply tell your device, in everyday language, what you want to do. And depending on how much information you choose to share with it, the software will be able to respond personally because it will have a rich understanding of your life. In the near future, anyone who’s online will be able to have a personal assistant powered by artificial intelligence that’s far beyond today’s technology.

This type of software—something that responds to natural language and can accomplish many different tasks based on its knowledge of the user—is called an agent. I’ve been thinking about agents for nearly 30 years and wrote about them in my 1995 book The Road Ahead, but they’ve only recently become practical because of advances in AI.

Agents are not only going to change how everyone interacts with computers. They’re also going to upend the software industry, bringing about the biggest revolution in computing since we went from typing commands to tapping on icons.

A personal assistant for everyone

Some critics have pointed out that software companies have offered this kind of thing before, and users didn’t exactly embrace them. (People still joke about Clippy, the digital assistant that we included in Microsoft Office and later dropped.) Why will people use agents?

The answer is that they’ll be dramatically better. You’ll be able to have nuanced conversations with them. They will be much more personalized, and they won’t be limited to relatively simple tasks like writing a letter. Clippy has as much in common with agents as a rotary phone has with a mobile device.

An agent will be able to help you with all your activities if you want it to. With permission to follow your online interactions and real-world locations, it will develop a powerful understanding of the people, places, and activities you engage in. It will get your personal and work relationships, hobbies, preferences, and schedule. You’ll choose how and when it steps in to help with something or ask you to make a decision.

“Clippy was a bot, not an agent.”

To see the dramatic change that agents will bring, let’s compare them to the AI tools available today. Most of these are bots. They’re limited to one app and generally only step in when you write a particular word or ask for help. Because they don’t remember how you use them from one time to the next, they don’t get better or learn any of your preferences. Clippy was a bot, not an agent.

Agents are smarter. They’re proactive—capable of making suggestions before you ask for them. They accomplish tasks across applications. They improve over time because they remember your activities and recognize intent and patterns in your behavior. Based on this information, they offer to provide what they think you need, although you will always make the final decisions.

Imagine that you want to plan a trip. A travel bot will identify hotels that fit your budget. An agent will know what time of year you’ll be traveling and, based on its knowledge about whether you always try a new destination or like to return to the same place repeatedly, it will be able to suggest locations. When asked, it will recommend things to do based on your interests and propensity for adventure, and it will book reservations at the types of restaurants you would enjoy. If you want this kind of deeply personalized planning today, you need to pay a travel agent and spend time telling them what you want.

The most exciting impact of AI agents is the way they will democratize services that today are too expensive for most people. They’ll have an especially big influence in four areas: health care, education, productivity, and entertainment and shopping.

Health care

Today, AI’s main role in healthcare is to help with administrative tasks. Abridge, Nuance DAX, and Nabla Copilot, for example, can capture audio during an appointment and then write up notes for the doctor to review.

The real shift will come when agents can help patients do basic triage, get advice about how to deal with health problems, and decide whether they need to seek treatment. These agents will also help healthcare workers make decisions and be more productive. (Already, apps like Glass Health can analyze a patient summary and suggest diagnoses for the doctor to consider.) Helping patients and healthcare workers will be especially beneficial for people in poor countries, where many never get to see a doctor at all.

These clinician-agents will be slower than others to roll out because getting things right is a matter of life and death. People will need to see evidence that health agents are beneficial overall, even though they won’t be perfect and will make mistakes. Of course, humans make mistakes too, and having no access to medical care is also a problem.

“Half of all U.S. military veterans who need mental health care don’t get it.”

Mental health care is another example of a service that agents will make available to virtually everyone. Today, weekly therapy sessions seem like a luxury. But there is a lot of unmet need, and many people who could benefit from therapy don’t have access to it. For example, RAND found that half of all U.S. military veterans who need mental health care don’t get it.

AI agents that are well trained in mental health will make therapy much more affordable and easier to get. Wysa and Youper are two of the early chatbots here. But agents will go much deeper. If you choose to share enough information with a mental health agent, it will understand your life history and your relationships. It’ll be available when you need it, and it will never get impatient. It could even, with your permission, monitor your physical responses to therapy through your smart watch—like if your heart starts to race when you’re talking about a problem with your boss—and suggest when you should see a human therapist.

Education

For decades, I’ve been excited about all the ways that software would make teachers’ jobs easier and help students learn. It won’t replace teachers, but it will supplement their work—personalizing the work for students and liberating teachers from paperwork and other tasks so they can spend more time on the most important parts of the job. These changes are finally starting to happen in a dramatic way.

The current state of the art is Khanmigo, a text-based bot created by Khan Academy. It can tutor students in math, science, and the humanities—for example, it can explain the quadratic formula and create math problems to practice on. It can also help teachers do things like write lesson plans. I’ve been a fan and supporter of Sal Khan’s work for a long time and recently had him on my podcast to talk about education and AI.

But text-based bots are just the first wave—agents will open up many more learning opportunities.

For example, few families can pay for a tutor who works one-on-one with a student to supplement their classroom work. If agents can capture what makes a tutor effective, they’ll unlock this supplemental instruction for everyone who wants it. If a tutoring agent knows that a kid likes Minecraft and Taylor Swift, it will use Minecraft to teach them about calculating the volume and area of shapes, and Taylor’s lyrics to teach them about storytelling and rhyme schemes. The experience will be far richer—with graphics and sound, for example—and more personalized than today’s text-based tutors.

Productivity

There’s already a lot of competition in this field. Microsoft is making its Copilot part of Word, Excel, Outlook, and other services. Google is doing similar things with Assistant with Bard and its productivity tools. These copilots can do a lot—such as turn a written document into a slide deck, answer questions about a spreadsheet using natural language, and summarize email threads while representing each person’s point of view.

Agents will do even more. Having one will be like having a person dedicated to helping you with various tasks and doing them independently if you want. If you have an idea for a business, an agent will help you write up a business plan, create a presentation for it, and even generate images of what your product might look like. Companies will be able to make agents available for their employees to consult directly and be part of every meeting so they can answer questions.

“If your friend just had surgery, your agent will offer to send flowers and be able to order them for you.”

Whether you work in an office or not, your agent will be able to help you in the same way that personal assistants support executives today. If your friend just had surgery, your agent will offer to send flowers and be able to order them for you. If you tell it you’d like to catch up with your old college roommate, it will work with their agent to find a time to get together, and just before you arrive, it will remind you that their oldest child just started college at the local university.

Entertainment and shopping

Already, AI can help you pick out a new TV and recommend movies, books, shows, and podcasts. Likewise, a company I’ve invested in, recently launched Pix, which lets you ask questions (“Which Robert Redford movies would I like and where can I watch them?”) and then makes recommendations based on what you’ve liked in the past. Spotify has an AI-powered DJ that not only plays songs based on your preferences but talks to you and can even call you by name.

Agents won’t simply make recommendations; they’ll help you act on them. If you want to buy a camera, you’ll have your agent read all the reviews for you, summarize them, make a recommendation, and place an order for it once you’ve made a decision. If you tell your agent that you want to watch Star Wars, it will know whether you’re subscribed to the right streaming service, and if you aren’t, it will offer to sign you up. And if you don’t know what you’re in the mood for, it will make customized suggestions and then figure out how to play the movie or show you choose.

You’ll also be able to get news and entertainment that’s been tailored to your interests. CurioAI, which creates a custom podcast on any subject you ask about, is a glimpse of what’s coming.

A shock wave in the tech industry

In short, agents will be able to help with virtually any activity and any area of life. The ramifications for the software business and for society will be profound.

In the computing industry, we talk about platforms—the technologies that apps and services are built on. Android, iOS, and Windows are all platforms. Agents will be the next platform.

“To create a new app or service, you’ll just tell your agent what you want.”

To create a new app or service, you won’t need to know how to write code or do graphic design. You’ll just tell your agent what you want. It will be able to write the code, design the look and feel of the app, create a logo, and publish the app to an online store. OpenAI’s launch of GPTs this week offers a glimpse into the future where non-developers can easily create and share their own assistants.

Agents will affect how we use software as well as how it’s written. They’ll replace search sites because they’ll be better at finding information and summarizing it for you. They’ll replace many e-commerce sites because they’ll find the best price for you and won’t be restricted to just a few vendors. They’ll replace word processors, spreadsheets, and other productivity apps. Businesses that are separate today—search advertising, social networking with advertising, shopping, productivity software—will become one business.

I don’t think any single company will dominate the agents business–there will be many different AI engines available. Today, agents are embedded in other software like word processors and spreadsheets, but eventually they’ll operate on their own. Although some agents will be free to use (and supported by ads), I think you’ll pay for most of them, which means companies will have an incentive to make agents work on your behalf and not an advertiser’s. If the number of companies that have started working on AI just this year is any indication, there will be an exceptional amount of competition, which will make agents very inexpensive.

But before the sophisticated agents I’m describing become a reality, we need to confront a number of questions about the technology and how we’ll use it. I’ve written before about the issues that AI raises, so I’ll focus specifically on agents here.

The technical challenges

Nobody has figured out yet what the data structure for an agent will look like. To create personal agents, we need a new type of database that can capture all the nuances of your interests and relationships and quickly recall the information while maintaining your privacy. We are already seeing new ways of storing information, such as vector databases, that may be better for storing data generated by machine learning models.

Another open question is about how many agents people will interact with. Will your personal agent be separate from your therapist agent and your math tutor? If so, when will you want them to work with each other and when should they stay in their lanes?

“If your agent needs to check in with you, it will speak to you or show up on your phone.”

How will you interact with your agent? Companies are exploring various options including apps, glasses, pendants, pins, and even holograms. All of these are possibilities, but I think the first big breakthrough in human-agent interaction will be earbuds. If your agent needs to check in with you, it will speak to you or show up on your phone. (“Your flight is delayed. Do you want to wait, or can I help rebook it?”) If you want, it will monitor sound coming into your ear and enhance it by blocking out background noise, amplifying speech that’s hard to hear, or making it easier to understand someone who’s speaking with a heavy accent.

There are other challenges too. There isn’t yet a standard protocol that will allow agents to talk to each other. The cost needs to come down so agents are affordable for everyone. It needs to be easier to prompt the agent in a way that will give you the right answer. We need to prevent hallucinations, especially in areas like health where accuracy is super-important, and make sure that agents don’t harm people as a result of their biases. And we don’t want agents to be able to do things they’re not supposed to. (Although I worry less about rogue agents than about human criminals using agents for malign purposes.)

Privacy and other big questions

As all of this comes together, the issues of online privacy and security will become even more urgent than they already are. You’ll want to be able to decide what information the agent has access to, so you’re confident that your data is shared with only people and companies you choose.

But who owns the data you share with your agent, and how do you ensure that it’s being used appropriately? No one wants to start getting ads related to something they told their therapist agent. Can law enforcement use your agent as evidence against you? When will your agent refuse to do something that could be harmful to you or someone else? Who picks the values that are built into agents?

There’s also the question of how much information your agent should share. Suppose you want to see a friend: If your agent talks to theirs, you don’t want it to say, “Oh, she’s seeing other friends on Tuesday and doesn’t want to include you.” And if your agent helps you write emails for work, it will need to know that it shouldn’t use personal information about you or proprietary data from a previous job.

Many of these questions are already top-of-mind for the tech industry and legislators. I recently participated in a forum on AI with other technology leaders that was organized by Sen. Chuck Schumer and attended by many U.S. senators. We shared ideas about these and other issues and talked about the need for lawmakers to adopt strong legislation.

But other issues won’t be decided by companies and governments. For example, agents could affect how we interact with friends and family. Today, you can show someone that you care about them by remembering details about their life—say, their birthday. But when they know your agent likely reminded you about it and took care of sending flowers, will it be as meaningful for them?

In the distant future, agents may even force humans to face profound questions about purpose. Imagine that agents become so good that everyone can have a high quality of life without working nearly as much. In a future like that, what would people do with their time? Would anyone still want to get an education when an agent has all the answers? Can you have a safe and thriving society when most people have a lot of free time on their hands?

But we’re a long way from that point. In the meantime, agents are coming. In the next few years, they will utterly change how we live our lives, online and off.

Source Gates Notes

Google Is Using Radar to Help Computers Read and React to Your Body Language

Posted by

0


Technology has quickly infiltrated almost every aspect of our lives, but the way we interface with our devices is still less than ideal. From hunching over our computer screens (because if you’re anything like me, it’s pretty much impossible to maintain good posture for more than a few minutes at a time) to constantly looking down at our phones (often while walking, driving, or otherwise being in motion), the way our bodies and brains interact with technology is not exactly seamless. Just how seamless we want it to become is debatable, but a Google project is exploring those boundaries.

Google’s Advanced Technology and Projects lab—ATAP—focuses on developing hardware to “change the way we relate to technology.” Its Project Jacquard developed conductive yarn to weave into clothing so people could interact with devices by, say, tapping their forearms—sort of like an elementary, fabric-based version of the Apple watch. The lab has also been working on a project called Soli, which uses radar to give computers spatial awareness, enabling them to interact with people non-verbally.

In other words, the project is trying to allow computers to recognize and respond to physical cues from their users, not unlike how we take in and respond to body language. “We are inspired by how people interact with one another,” said Leonardo Giusti, ATAP’s Head of Design. “As humans, we understand each other intuitively, without saying a single word. We pick up on social cues, subtle gestures that we innately understand and react to. What if computers understood us this way?”

Examples include a computer automatically powering up when you get within a certain distance of it or pausing a video when you look away from the screen.

The sensor works by sending out electromagnetic waves in a broad beam, which are intercepted and reflected back to the radar antenna by objects—or people—in their path. The reflected waves are analyzed for properties like energy, time delay, and frequency shift, which give clues about the reflector’s size, shape, and distance from the sensor. Parsing the data even further using a machine learning algorithm enables the sensor to determine things like an object’s orientation, its distance from the device, and the speed of its movements.

The ATAP team helped train Soli’s algorithm themselves by doing a series of movements while being tracked by cameras and radar sensors. The movements they focused on were ones typically involved in interacting with digital devices, like turning toward or away from a screen, approaching or leaving a space or device, glancing at a screen, etc. The ultimate goal is for the sensor to be able to anticipate a user’s next move and serve up a corresponding response, facilitating the human-device interaction by enabling devices to “understand the social context around them,” as ATAP’s Human-Computer Interaction Lead Eiji Hayashi put it.

Improving the way we interact with our now-ubiquitous devices isn’t a new idea. Jody Medich, principal design researcher at Microsoft and CEO of Superhuman-X, has long been advocating for what she calls human-centered technology, maintaining that our interfaces “are killing our ability to think” by overloading our working memory (which is short-term and task-based) with constant interruptions.

In 2017 Medich predicted the rise of perceptual computing, in which machines recognize what’s happening around them and act accordingly. “This will cause the dematerialization curve to dramatically accelerate while we use technology in even more unexpected locations,” she wrote. “This means technology will be everywhere, and so will interface.”

It seems she wasn’t wrong, but this begs a couple of important questions.

First, do we really need our computers to “understand” and respond to our movements and gestures? Is this a necessary tweak to how we use technology, or a new apex of human laziness? Pressing pause on a video before getting up to walk away takes a split second, as does pressing the power button to turn a device on or off. And what about those times we want the computer to stay on or the video to keep playing even when we’re not right in front of the screen?

Secondly, what might the privacy implications of these sensor-laden devices be? The ATAP team emphasizes that Soli uses radar precisely because it protects users’ privacy far more than, say, cameras; radar can’t distinguish between different peoples’ faces or bodies, it can just tell that there’s a person in its space. Also, data from the Soli sensor in Google’s Nest Hub doesn’t get sent to the cloud, it’s only processed locally on users’ devices, and the assumption is that a product made for laptops or other devices would function the same way.

People may initially be creeped out by their devices being able to anticipate and respond to their movements. Like most other technology we initially find off-putting for privacy reasons, though, it seems we ultimately end up valuing the convenience these products give us more than we value our privacy; it all comes down to utilitarianism.

Whether or not we want our devices to eventually become more like extensions of our bodies, it’s likely the technology will move in that direction. Analyses from 2019 through this year estimate we check our phones anywhere from 96 to 344 times per day. That’s a lot of times, and a lot of interrupting what we’re doing to look at these tiny screens that now essentially run our lives.

Is there a better way? Hopefully. Is this it? TBD.

Why Do Computers Use So Much Energy?

Posted by

0


It’s possible they could be vastly more efficient, but for that to happen we need to better understand the thermodynamics of computing

Why Do Computers Use So Much Energy?

Microsoft is currently running an interesting set of hardware experiments. The company is taking a souped-up shipping container stuffed full of computer servers and submerging it in the ocean. The most recent round is taking place near Scotland’s Orkney Islands, and involves a total of 864 standard Microsoft data-center servers. Many people have impugned the rationality of the company that put Seattle on the high-tech map, but seriously—why is Microsoft doing this?

There are several reasons, but one of the most important is that it is far cheaper to keep computer servers cool when they’re on the seafloor. This cooling is not a trivial expense. Precise estimates vary, but currently about 5 percent of all energy consumption in the U.S. goes just to running computers—a huge cost to the economy as whole. Moreover, all that energy used by those computers ultimately gets converted into heat. This results in a second cost: that of keeping the computers from melting.

These issues don’t only arise in artificial, digital computers. There are many naturally occurring computers, and they, too, require huge amounts of energy. To give a rather pointed example, the human brain is a computer. This particular computer uses some 10–20 percent of all the calories that a human consumes. Think about it: our ancestors on the African savanna had to find 20 percent more food every single day, just to keep that ungrateful blob of pink jelly imperiously perched on their shoulders from having a hissy fit. That need for 20 percent more food is a massive penalty to the reproductive fitness of our ancestors. Is that penalty why intelligence is so rare in the evolutionary record? Nobody knows—and nobody has even had the mathematical tools to ask the question before.

There are other biological computers besides brains, and they too consume large amounts of energy. To give one example, many cellular systems can be viewed as computers. Indeed, the comparison of thermodynamic costs in artificial and cellular computers can be extremely humbling for modern computer engineers. For example, a large fraction of the energy budget of a cell goes to translating RNA into sequences of amino acids (i.e., proteins), in the cell’s ribosome. But the thermodynamic efficiency of this computation—the amount of energy required by a ribosome per elementary operation—is many orders of magnitude superior to the thermodynamic efficiency of our current artificial computers. Are there “tricks” that cells use that we could exploit in our artificial computers? Going back to the previous biological example, are there tricks that human brains use to do their computations that we can exploit in our artificial computers?

More generally, why do computers use so much energy in the first place? What are the fundamental physical laws governing the relationship between the precise computation a system runs and how much energy it requires? Can we make our computers more energy-efficient by redesigning how they implement their algorithms?

These are some of the issues my collaborators and I are grappling with in an ongoing research project at the Santa Fe Institute. We are not the first to investigate these issues; they have been considered, for over a century and a half, using semi-formal reasoning based on what was essentially back-of-the-envelope style analysis rather than rigorous mathematical arguments—since the relevant math wasn’t fully mature at the time.

This earlier work resulted in many important insights, in particular the work in the mid to late 20th century by Rolf Landauer, Charles Bennett and others.

However, this early work was also limited by the fact that it tried to apply equilibrium statistical physics to analyze the thermodynamics of computers. The problem is that, by definition, an equilibrium system is one whose state never changes. So whatever else they are, computers are definitely nonequilibrium systems.  In fact, they are often very-far-from-equilibrium systems.

Fortunately, completely independent of this early work, there have been some major breakthroughs in the past few decades in the field of nonequilibrium statistical physics (closely related to a field called “stochastic thermodynamics”). These breakthroughs allow us to analyze all kinds of issues concerning how heat, energy, and information get transformed in nonequilibrium systems.

These analyses have provided some astonishing predictions. For example, we can now calculate the (non-zero) probability that a given nanoscale system will violate the second law, reducing its entropy, in a given time interval. (We now understand that the second law does not say that the entropy of a closed system cannot decrease, only that its expected entropy cannot decrease.) There are no controversies here arising from semi-formal reasoning; instead, there are many hundreds of peer-reviewed articles in top journals, a large fraction involving experimental confirmations of theoretical predictions.

Now that we have the right tools for the job, we can revisit the entire topic of the thermodynamics of computation in a fully formal manner. This has already been done for bit erasure, the topic of concern to Landauer and others, and we now have a fully formal understanding of the thermodynamic costs in erasing a bit (which turn out to be surprisingly subtle).

However, computer science extends far, far beyond counting the number of bit erasures in a given computation. Thanks to the breakthroughs of nonequilibrium statistical physics, we can now also investigate the rest of computer science from a thermodynamic perspective. For example, moving from bits to circuits, my collaborators and I now have a detailed analysis of the thermodynamic costs of “straight-line circuits.” Surprisingly, this analysis has resulted in novel extensions of information theory. Moreover, in contrast to the kind of analysis pioneered by Landauer, this analysis of the thermodynamic costs of circuits is exact, not just a lower bound.

Conventional computer science is about all about trade-offs between the memory resources and number of timesteps needed to perform a given computation. In light of the foregoing, it seems that there might be far more thermodynamic trade-offs in performing a computation than had been appreciated in conventional computer science, involving thermodynamic costs in addition to the costs of memory resources and number of timesteps. Such trade-offs would apply in both artificial and biological computers.

Clearly there is a huge amount to be done to develop this modern “thermodynamics of computation.”

Be on the lookout for a forthcoming book from the SFI Press, of contributed papers touching on many of the issues mentioned above. Also, to foster research on this topic we have built a wiki, combining lists of papers, websites, events pages, etc. We highly encourage people to visit it, sign up, and start improving it; the more scientists get involved, from the more fields, the better!

Why Do Computers Use So Much Energy?

Posted by

0


It’s possible they could be vastly more efficient, but for that to happen we need to better understand the thermodynamics of computing

Why Do Computers Use So Much Energy?
Microsoft is currently running an interesting set of hardware experiments. The company is taking a souped-up shipping container stuffed full of computer servers and submerging it in the ocean. The most recent round is taking place near Scotland’s Orkney Islands, and involves a total of 864 standard Microsoft data-center servers. Many people have impugned the rationality of the company that put Seattle on the high-tech map, but seriously—why is Microsoft doing this?

There are several reasons, but one of the most important is that it is far cheaper to keep computer servers cool when they’re on the seafloor. This cooling is not a trivial expense. Precise estimates vary, but currently about 5 percent of all energy consumption in the U.S. goes just to running computers—a huge cost to the economy as whole. Moreover, all that energy used by those computers ultimately gets converted into heat. This results in a second cost: that of keeping the computers from melting.

These issues don’t only arise in artificial, digital computers. There are many naturally occurring computers, and they, too, require huge amounts of energy. To give a rather pointed example, the human brain is a computer. This particular computer uses some 10–20 percent of all the calories that a human consumes. Think about it: our ancestors on the African savanna had to find 20 percent more food every single day, just to keep that ungrateful blob of pink jelly imperiously perched on their shoulders from having a hissy fit. That need for 20 percent more food is a massive penalty to the reproductive fitness of our ancestors. Is that penalty why intelligence is so rare in the evolutionary record? Nobody knows—and nobody has even had the mathematical tools to ask the question before.

There are other biological computers besides brains, and they too consume large amounts of energy. To give one example, many cellular systems can be viewed as computers. Indeed, the comparison of thermodynamic costs in artificial and cellular computers can be extremely humbling for modern computer engineers. For example, a large fraction of the energy budget of a cell goes to translating RNA into sequences of amino acids (i.e., proteins), in the cell’s ribosome. But the thermodynamic efficiency of this computation—the amount of energy required by a ribosome per elementary operation—is many orders of magnitude superior to the thermodynamic efficiency of our current artificial computers. Are there “tricks” that cells use that we could exploit in our artificial computers? Going back to the previous biological example, are there tricks that human brains use to do their computations that we can exploit in our artificial computers?

More generally, why do computers use so much energy in the first place? What are the fundamental physical laws governing the relationship between the precise computation a system runs and how much energy it requires? Can we make our computers more energy-efficient by redesigning how they implement their algorithms?

These are some of the issues my collaborators and I are grappling with in an ongoing research project at the Santa Fe Institute. We are not the first to investigate these issues; they have been considered, for over a century and a half, using semi-formal reasoning based on what was essentially back-of-the-envelope style analysis rather than rigorous mathematical arguments—since the relevant math wasn’t fully mature at the time.

This earlier work resulted in many important insights, in particular the work in the mid to late 20th century by Rolf Landauer, Charles Bennett and others.

However, this early work was also limited by the fact that it tried to apply equilibrium statistical physics to analyze the thermodynamics of computers. The problem is that, by definition, an equilibrium system is one whose state never changes. So whatever else they are, computers are definitely nonequilibrium systems.  In fact, they are often very-far-from-equilibrium systems.

Fortunately, completely independent of this early work, there have been some major breakthroughs in the past few decades in the field of nonequilibrium statistical physics (closely related to a field called “stochastic thermodynamics”). These breakthroughs allow us to analyze all kinds of issues concerning how heat, energy, and information get transformed in nonequilibrium systems.

These analyses have provided some astonishing predictions. For example, we can now calculate the (non-zero) probability that a given nanoscale system will violate the second law, reducing its entropy, in a given time interval. (We now understand that the second law does not say that the entropy of a closed system cannot decrease, only that its expected entropy cannot decrease.) There are no controversies here arising from semi-formal reasoning; instead, there are many hundreds of peer-reviewed articles in top journals, a large fraction involving experimental confirmations of theoretical predictions.

Now that we have the right tools for the job, we can revisit the entire topic of the thermodynamics of computation in a fully formal manner. This has already been done for bit erasure, the topic of concern to Landauer and others, and we now have a fully formal understanding of the thermodynamic costs in erasing a bit (which turn out to be surprisingly subtle).

However, computer science extends far, far beyond counting the number of bit erasures in a given computation. Thanks to the breakthroughs of nonequilibrium statistical physics, we can now also investigate the rest of computer science from a thermodynamic perspective. For example, moving from bits to circuits, my collaborators and I now have a detailed analysis of the thermodynamic costs of “straight-line circuits.” Surprisingly, this analysis has resulted in novel extensions of information theory. Moreover, in contrast to the kind of analysis pioneered by Landauer, this analysis of the thermodynamic costs of circuits is exact, not just a lower bound.

Conventional computer science is about all about trade-offs between the memory resources and number of timesteps needed to perform a given computation. In light of the foregoing, it seems that there might be far more thermodynamic trade-offs in performing a computation than had been appreciated in conventional computer science, involving thermodynamic costs in addition to the costs of memory resources and number of timesteps. Such trade-offs would apply in both artificial and biological computers.

Clearly there is a huge amount to be done to develop this modern “thermodynamics of computation.”

Be on the lookout for a forthcoming book from the SFI Press, of contributed papers touching on many of the issues mentioned above. Also, to foster research on this topic we have built a wiki, combining lists of papers, websites, events pages, etc. We highly encourage people to visit it, sign up, and start improving it; the more scientists get involved, from the more fields, the better!

Study suggests a direct link between screen time and ADHD in teens

Posted by

0


Image: Study suggests a direct link between screen time and ADHD in teens

Adding to the list of health concerns associated with excessive screen time, one study suggests that there could be a link between the length of time teenagers spend online and attention deficit hyperactivity disorder (ADHD).

The two-year study, which was published in the Journal of the American Medical Association (JAMA), observed more than 2,500 high school students from Los Angeles.

Digital media and the attention span of teenagers

A team of researchers analyzed data from the teenagers who had shorter attention spans the more they became involved in different digital media platforms for the duration of the experiment.

The JAMA study observed adolescents aged 15 and 16 years periodically for two years. The researchers asked the teenagers about the frequency of their online activities and if they had experienced any of the known symptoms of ADHD.

As the teenagers’ digital engagement rose, their reported ADHD symptoms also went up by 10 percent. The researchers noted that based on the results of the study, even if digital media usage does not definitively cause ADHD, it could cause symptoms that would result in the diagnosis of ADHD or require pharmaceutical treatment.

Experts believe that ADHD begins in the early stages of childhood development. However, the exact circumstances, regardless if they are biological or environmental, have yet to be determined.

Adam Leventhal, a University of Southern California psychologist and senior author of the study, shared that the research team is now analyzing the occurrence of new symptoms that were not present when the study began.

The power of the elements: Discover Colloidal Silver Mouthwash with quality, natural ingredients like Sangre de Drago sap, black walnut hulls, menthol crystals and more. Zero artificial sweeteners, colors or alcohol. Learn more at the Health Ranger Store and help support this news site.

Other studies about digital engagement have implied that there is an inverse relationship with happiness. The less people used digital media, the more they reported feeling an overall sense of happiness. (Related: The social media paradox: Teens who are always online feel more lonely.)

The researchers concluded that the teenagers might have exhibited ADHD symptoms from the outset due to other factors. However, it is possible that excessive digital media usage can still aggravate these symptoms.

Fast facts about ADHD

ADHD is a neurodevelopmental disorder that is commonly diagnosed in children. However, it can also be diagnosed in older individuals. ADHD can be difficult to diagnose. Since several symptoms of ADHD are similar to normal childhood behaviors, the disorder itself can be hard to detect.

The symptoms of ADHD may include forgetting completed tasks, having difficulty sitting still, having difficulty staying organized, and having trouble concentrating or focusing.

  • Men are at least three times more likely to be diagnosed with ADHD than females.
  • During their lifetimes, at least 13 percent of men will be diagnosed with ADHD, as opposed to only 4.2 percent in women.
  • The average age of ADHD diagnosis is seven years old.
  • The symptoms of the condition will usually manifest when a child is aged three to six years old.
  • ADHD is not solely a childhood disorder. At least four percent of American adults older than 18 may have ADHD.

This disorder does not increase an individual’s risk for other conditions or diseases. However, some people with ADHD, mostly children, have a higher chance of experiencing different coexisting conditions. These can make social situations, like school, more difficult for kids with ADHD.

Some coexisting conditions of ADHD may include:

  • Anxiety disorder
  • Bed-wetting problems
  • Bipolar disorder
  • Conduct disorders and difficulties (e.g., antisocial behavior, fighting, and oppositional defiant disorder)
  • Depression
  • Learning disabilities
  • Sleep disorders
  • Substance abuse
  • Tourette syndrome

Minimize your child’s ADHD risk by reading more articles with tips on how to manage their internet use at Addiction.news.

Sources include:

Lifezette.com

Healthline.com

Thanks to AI, Computers Can Now See Your Health Problems. 

Posted by

0


PATIENT NUMBER TWO was born to first-time parents, late 20s, white. The pregnancy was normal and the birth uncomplicated. But after a few months, it became clear something was wrong. The child had ear infection after ear infection and trouble breathing at night. He was small for his age, and by his fifth birthday, still hadn’t spoken. He started having seizures. Brain MRIs, molecular analyses, basic genetic testing, scores of doctors; nothing turned up answers. With no further options, in 2015 his family decided to sequence their exomes—the portion of the genome that codes for proteins—to see if he had inherited a genetic disorder from his parents. A single variant showed up: ARID1B.

The mutation suggested he had a disease called Coffin-Siris syndrome. But Patient Number Two didn’t have that disease’s typical symptoms, like sparse scalp hair and incomplete pinky fingers. So, doctors, including Karen Gripp, who met with Two’s family to discuss the exome results, hadn’t really considered it. Gripp was doubly surprised when she uploaded a photo of Two’s face to Face2Gene. The app, developed by the same programmers who taught Facebook to find your face in your friend’s photos, conducted millions of tiny calculations in rapid succession—how much slant in the eye? How narrow is that eyelid fissure? How low are the ears? Quantified, computed, and ranked to suggest the most probable syndromes associated with the facial phenotype. There’s even a heat map overlay on the photo that shows which the features are the most indicative match.

“In hindsight it was all clear to me,” says Gripp, who is chief of the Division of Medical Genetics at A.I. duPont Hospital for Children in Delaware, and had been seeing the patient for years. “But it hadn’t been clear to anyone before.” What had taken Patient Number Two’s doctors 16 years to find took Face2Gene just a few minutes.

Face2Gene takes advantage of the fact that so many genetic conditions have a tell-tale “face”—a unique constellation of features that can provide clues to a potential diagnosis. It is just one of several new technologies taking advantage of how quickly modern computers can analyze, sort, and find patterns across huge reams of data. They are built in fields of artificial intelligence known as deep learning and neural nets—among the most promising to deliver AI’s 50-year old promise to revolutionize medicine by recognizing and diagnosing disease.

 Genetic syndromes aren’t the only diagnoses that could get help from machine learning. The RightEye GeoPref Autism Test can identify the early stages of autism in infants as young as 12 months—the crucial stages where early intervention can make a big difference. Unveiled January 2 at CES in Las Vegas, the technology uses infrared sensors test the child’s eye movement as they watch a split-screen video: one side fills with people and faces, the other with moving geometric shapes. Children at that age should be much more attracted to faces than abstract objects, so the amount of time they look at each screen can indicate where on the autism spectrum a child might fall.

In validation studies done by the test’s inventor, UC San Diego researcher Karen Pierce,1the test correctly predicted autism spectrum disorder 86 percent of the time in more than 400 toddlers. That said, it’s still pretty new, and hasn’t yet been approved by the FDA as a diagnostic tool. “In terms of machine learning, it’s the simplest test we have,” says RightEye’s Chief Science Officer Melissa Hunfalvay. “But before this, it was just physician or parent observations that might lead to a diagnosis. And the problem with that is it hasn’t been quantifiable.”

A similar tool could help with early detection of America’s sixth leading cause of death: Alzheimer’s disease. Often, doctors don’t recognize physical symptoms in time to try any of the disease’s few existing interventions. But machine learning hears what doctor’s can’t: Signs of cognitive impairment in speech. This is how Toronto-based Winterlight Labs is developing a tool to pick out hints of dementia in its very early stages. Co-founder Frank Rudzicz calls these clues “jitters,” and “shimmers:” high frequency wavelets only computers, not humans, can hear.

Winterlight’s tool is way more sensitive than the pencil and paper-based tests doctor’s currently use to assess Alzheimer’s. Besides being crude, data-wise, those tests can’t be taken more than once every six months. Rudzicz’s tool can be used multiple times a week, which lets it track good days, bad days, and measure a patient’s cognitive functions over time. The product is still in beta, but is currently being piloted by medical professionals in Canada, the US, and France.

If this all feels a little scarily sci-fi to you, it’s useful to remember that doctors have been trusting computers with your diagnoses for a long time. That’s because machines are much more sensitive at both detecting and analyzing the many subtle indications that our bodies are misbehaving. For instance, without computers, Patient Number Two would never have been able to compare his exome to thousands of others, and find the genetic mutation marking him with Coffin-Siris syndrome.

But none of this makes doctors obsolete. Even Face2Gene—which, according to its inventors, can diagnose up to half of the 8,000 known genetic syndromes using facial patterns gleaned from the hundreds of thousands of images in its database—needs a doctor (like Karen Gripp) with enough experience to verify the results. In that way, machines are an extension of what medicine has always been: A science that grows more powerful with every new data point.

Computers Made of Genetic Material Will Revolutionize Our World

Posted by

0


IN BRIEF
  • Researchers have been able to create tiny structures for conducting electricity by using DNA and gold plating.
  • This new nanostructure could be the foundation of future electronics as soon as improvements are made on this breakthrough development.

GOLD AND DNA

Nanostructures made using DNA origami are fascinating. The ability to use DNA as a construction material, capable of holding scaffolds of molecules and atoms was one huge step in developing modern nanostrutures. Most recent of these developments are gold-plated nanowires constructed by scientists from the Helmholtz-Zentrum Dresden-Rossendorf (HZDR) and from Paderborn University, which independently assembled themselves from single DNA strands, as published in the journal Langmuir.

These nanowires, due to their gold-plating, were able to conduct electricity. “Our measurements have shown that an electrical current is conducted through these tiny wires,” explains Artur Erbe of the Institute of Ion Beam Physics and Materials Research. The nano-sized structures were connected by two electrical contacts.

Even more fascinating is how these were made using modified DNA strands — stable double strands combined through their base pairs, from long single strands of genetic material and DNA segments. These allowed for the structures to independently take on their desired forms, complex structures developed by molecules through a self-assembling processes.

FROM THE BOTTOM-UP

“With the help of this approach, which resembles the Japanese paper folding technique origami and is therefore referred to as DNA-origami, we can create tiny patterns. Extremely small circuits made of molecules and atoms are also conceivable here,” says Erbe.

Usually, developing nano circuits use what is known as the “top-down” method, where the base material is chiseled until the desired structure is formed. This will become increasingly difficult as electronics continue miniaturization. The new “bottom-up” method changes how these electronic components are usually made.

Credits: B. Teschome, A. Erbe, et al.
Credits: B. Teschome, A. Erbe, et al.

There is one problem, though. “Genetic matter doesn’t conduct a current particularly well,” Erbe points out, which explains why the nanowires were gold-plated. But even with this, there was still difficulty with conducting current at room temperatures. Better melding of conductive materials need to be further developed, plus the option of using cheaper, more standard wire coating than gold.

Still, the research is promising. This nanowire that’s made partially out of genetic material could be the future of electronics. Smaller wires allow for more compact designs, which together with smaller transistors, can be used to make more powerful computers.

Will Computers Replace Radiologists?

Posted by

0


I recently told a radiology resident who demolished the worklist, “You’re a machine.” He beamed with pride. Imitation is the highest form of flattery. But the machine, not content in being imitated, wants to encroach on our turf.

CT could scarcely have progressed without progress in computing to horde the glut of thin slices. On two-dimensional projectional images such as abdominal radiographs, arteries reveal themselves only if calcified. With CT, algorithms extract arteries with a click. When I first saw this I was mesmerized, then humbled, and then depressed. With automation, what was my role, aside from clicking the aorta?

The role of computers in radiology was predicted as early as 1959 by Lee Lusted,[1] a radiologist and the founder of the Society for Medical Decision Making. Lusted envisioned “an electronic scanner-computer to look at chest photofluorograms, to separate the clearly normal chest films from the abnormal chest films. The abnormal chest films would be marked for later study by the radiologists.”

AI Is Not Just a Sci-Fi Film

I was skeptical of the machine. Skepticism is ignorance waiting for enlightenment. Nearly 60 years after Lusted’s intuition, artificial intelligence (AI) is not a figment of Isaac Asimov’s imagination but a virtual reality.

Leading the way in AI is IBM. Tanveer Syeda-Mahmood, PhD, IBM’s chief scientist for the Medical Sieve Radiology Grand Challenge project, sees symbiosis between radiologists and AI. At the 2015 RSNA meeting she showed off Watson, IBM’s polymath and the poster child of AI. Watson can find clots in the pulmonary arteries. The quality of the demonstration cases was high; the pulmonary arteries shone brightly. But I’m being nitpicky pointing that out. Evolutionarily, radiologists have reached their apotheosis. But Watson will improve. He has a lot of homework—30 billion medical images to review after IBM acquired Merge, according to the Wall Street Journal.[2] A busy radiologist reads approximately 20,000 studies a year.

AI has a comparative advantage over radiologists because of the explosion of images: In a trauma patient, a typical “pan scan”—CT of the head, cervical spine, chest, abdomen, and pelvis with reformats—renders over 4000 images. Aside from the risk for de Quervain’s tenosynovitis from scrolling through the images, radiologists face visual fatigue. Our visual cortex was designed to look for patterns, not needles in haystacks.[3]

The explosion of images has led to an odd safety concern. The disease burden hasn’t risen, and the prevalence of clinically significant pathology on imaging remains the same; however, the presence ofpotentially significant pathology has increased and the chances of missing potentially significant pathology have increased exponentially—ie, there is an epidemic of possible medical errors. Radiologists will be no match for AI in detecting 1-mm lung nodules.

What’s intriguing about AI is what AI finds easy. While I find the beating heart demanding to stare at, cine imaging is a piece of cake for Watson. Watson learned about cardiac disease by seeing videotapes of echocardiograms from India, according to Syeda-Mahmood. True to the healthcare systems that Watson will serve, he has both American and foreign training.

What’s even more intriguing about AI is what AI finds difficult. Eliot Siegel, MD, professor of radiology at the University of Maryland, has promised me that if anyone develops a program that segments the adrenal glands on CT as reliably as a 7-year-old could, he’d wash their car. So far he hasn’t had to wash any cars. AI may crunch PE studies but struggle with intra-abdominal abscesses. Distinguishing between fluid-filled sigmoid colon and an infected collection isn’t the last frontier of technology, but it may be the last bastion of the radiologist.

Competition for Watson

Watson has a rival. Enlitic was founded by Australian data scientist and entrepreneur Jeremy Howard. Both Watson and Enlitic use deep learning, an unregulated process whereby the computer figures out why something is what it is after being shown several examples. However, their philosophy is different. Watson wants to understand the disease. Enlitic wants to understand the raw data. Enlitic’s philosophy is a scientific truism: Images = f (x). Imaging is data. Find the source data, solve the data, and you’ve solved the diagnosis.

Igor Barani, MD, a radiation oncologist and the CEO of Enlitic, told me that he was once skeptical of computers. He changed his mind when he saw what Enlitic could do. After being fed several hundred musculoskeletal radiographs, which were either normal or had fractures, the machine flagged not only the radiographs with fracture but also the site of the fracture. The machine started off ignorant, was not told what to do, and learned by trial and error. It wasn’t spoonfed— rather, it feasted. Enlitic is like that vanishingly uncommon autodidact.

Barani, too, believes that AI and radiologists are symbiotic. According to him, AI will not render radiologists unemployable but will save the radiologist from mundane tasks that can be automated, such as reading portable chest radiographs to confirm line placements and looking at CT scans for lung nodules. As he put it, “You did not go to medical school to measure lung nodules.” Barani’s point is well taken. The tasks that can be automated should be given to the machine—not as surrender but secession.

Medical Publications vs Business News

Living in San Francisco, Barani is aware of the hot air from Silicon Valley. He doesn’t want Enlitic to be cast in the same mold as some diagnostics that have been buried under their own hype. He wants the technology to prove itself in a randomized controlled trial and says that Enlitic is conducting such a trial in Australia, which will assess AI’s accuracy and efficiency as an adjunct to radiologists. This is just as well. Charles Kahn, MD, an expert in informatics and vice chair of radiology at the University of Pennsylvania, has followed the history of neural networks—AI’s DNA. “I’ve seen optimism before. It’s time that proponents of AI published, not in Forbes, but in peer-reviewed medical journals,” he told me, with slight exasperation.

For AI’s full potential to be harnessed, it must extract as much information as possible from the electronic health record (EHR), images, and radiology reports. The challenges are not computational. The first challenge is the validity of the information. For example, the EHR has signal, but it also has a lot of junk that looks like signal because an ICD-10 code is attached.

The second challenge is consistency. The variability and disclaimers in radiology reports that frustrate clinicians could frustrate AI as well. It needs a consistent diagnostic anchor. Imagine if half of the World Atlases listed the capital of Mongolia as Ulan Bator and the other half listed it as New Delhi. Even Watson might be confused if asked the capital of Mongolia.

Could the hedge “pneumonia not excluded”—the Achilles heel of radiologists, the chink in the radiologist’s armor—save it from AI? Gleefully, I asked IBM’s Syeda-Mahmood. She smiled. “Watson doesn’t need to be better than you. Just as good as you.” She has a point. If Watson knows to pick up emboli in the lobar pulmonary arteries in some studies and can report in others “no embolus seen but subsegmental pulmonary embolus is not excluded,” how is that different from what we do?

Digital Mammography and AI Takeover

Like radiologists, AI must choose between sensitivity and specificity—ie, between overcalling and undercalling disease. One imperative for computer assistance in diagnosis is to reduce diagnostic errors. The errors considered more egregious are misses, not overdiagnoses. AI will favor sensitivity over specificity.

 If AI reminds radiologists that leiomyosarcoma of the pulmonary veins, for example, is in the differential for upper lobe blood diversion on chest radiograph, this rare neoplasm will never be missed. But there’s a fine line between being a helpful Siri and a monkey on the shoulder constantly crying wolf.

The best example of computer-aided detection (CAD) is in breast imaging. Touted to revolutionize mammography, CAD’s successes have been modest.[4] CAD chose sensitivity over specificity in a field where false negatives are dreaded, false positives cost, and images are complex. CAD flags several pseudoabnormalities, which experienced readers summarily dismiss but over which novice readers ruminate. CAD has achieved neither higher sensitivity nor higher specificity.

When I asked Emily Conant, MD, chief of breast imaging at the University of Pennsylvania, about this, she cautioned against early dismissal of CAD for mammography. “With digital platforms, quantification of global measures of tissue density and complexity are being developed to aid detection. CAD will be more reproducible and efficient than human readers in quantifying. This will be a great advance.” Digital mammography follows a pattern seen in other areas of imaging. An explosion of information is followed by an imperative to quantify, leaving radiologists vulnerable to annexation by the machine.

Should radiologists view AI as a friend or a foe? The question is partly moot. If AI has a greater area under the receiver operating character curve than radiologists—meaning it calls fewer false negativesand fewer false positives—it hardly matters what radiologists feel. Progress in AI will be geometric. Once all data are integrated, AI can have a greater sensitivity and greater specificity than radiologists.

Do We Need Fewer Radiologists?

Workforce planning for organized radiology is tricky. That AI will do the job radiologists do today is a mathematical certainty. The question is when. If it were within 6 months, radiologists may as well fall on their swords today. A reasonable timeframe is anything between 10 and 40 years, but closer to 10 years. How radiologists and AI could interact might be beyond our imagination. Enlitic’s Barani believes that radiologists can use AI to look after populations. AI, he says, “can scale the locus of a radiologist’s influence.”

AI may increase radiologists’ work in the beginning as it spits out false positives to dodge false negatives. I consulted R. Nick Bryan, MD, PhD, emeritus professor at the University of Pennsylvania, who believes that radiologists will adjudicate normal. The arc of history bends toward irony. Lusted thought that computers would find normal studies, leaving abnormal ones for radiologists. The past belonged to sensitivity; the future is specificity. People are tired of being told that they have “possible disease.” They want to know if they’re normal.

Bryan, a neuroradiologist, founded a technology that uses Bayesian analysis and a library of reference images for diagnosis in neuroimaging. He claims that the technology does better than first-year radiology residents and as well as neuroradiology fellows in correctly describing a range of brain diseases on MRI. He once challenged me to a duel with the technology. I told him that I was washing my hair that evening.

Running from AI isn’t the solution. Radiologists must work with AI, not to improve AI but to realize their role in the post-AI information world. Radiologists must keep their friends close and AI closer. Automation affects other industries. We have news stories written by bots (standardized William Zinsser–inspired op-eds may be next). Radiologists shouldn’t take automation personally.

Nevertheless, radiologists must know themselves. Emmanuel Botzolakis, MD, a neuroradiology fellow working with Bryan, put it succinctly. “Radiologists should focus on inference, not detection. With detection, we’ll lose to AI. With inference, we might prevail.”

Botzolakis was distinguishing between Radiologist as Clinical Problem Solver and Radiologist as TSA Detector. Like TSA detectors, radiologists spot possible time bombs, which on imaging are mostly irrelevant to the clinical presentation. This role is not likely to diminish, because there will be more anticipatory medicine and more quantification. When AI becomes that TSA detector, we may need fewer radiologists per capita to perform more complex cognitive tasks.

The future belongs to quantification but it is far from clear that this future will be palatable to its users. AI could attach numerical probabilities to differential diagnosis and churn reports such as this:

Based on Mr Patel’s demographics and imaging, the mass in the liver has a 66.6% chance of being benign, 33.3% chance of being malignant, and a 0.1% of not being real.

Is precision as useful as we think? What will you do if your CT report says renal mass has a “0.8% chance of malignancy?” Sleep soundly? Remove the mass? Do follow-up imaging? Numbers are continuous but decision-making is dichotomous, and the final outcome is still singular. Is the human race prepared for all of this information?

The hallmark of intelligence is in reducing information to what is relevant. A dualism may emerge between artificial and real intelligence, where AI spits out information and radiologists contract information. Radiologists could be Sherlock Holmes to the untamed eagerness of Watson.

In the meantime, radiologists should ask which tasks need a medical degree. Surely, placing a caliper from one end of a lung nodule to the other end doesn’t need 4 years of medical school and an internship. Then render unto AI what is AI’s. Of all the people I spoke to for this article, Gregory Mogel, MD, a battle-hardened radiologist and chief of radiology at Central Valley Kaiser Permanente, said it best. “Any radiologist that can be replaced by a computer should be.” Amen.

 

Will computers ever truly understand what we’re saying?

Posted by

0


Will computers ever truly understand what we're saying?
As two people conversing rely more and more on previously shared concepts, the same area of their brains — the right superior temporal gyrus — becomes more active (blue is activity in communicator, orange is activity in interpreter). This suggests that this brain region is key to mutual understanding as people continually update their shared understanding of the context of the conversation to improve mutual understanding. Credit: Arjen Stolk, UC Berkeley

From Apple’s Siri to Honda’s robot Asimo, machines seem to be getting better and better at communicating with humans.

But some neuroscientists caution that today’s computers will never truly understand what we’re saying because they do not take into account the context of a conversation the way people do.

Specifically, say University of California, Berkeley, postdoctoral fellow Arjen Stolk and his Dutch colleagues, machines don’t develop a shared understanding of the people, place and situation – often including a long social history – that is key to human . Without such common ground, a computer cannot help but be confused.

“People tend to think of communication as an exchange of linguistic signs or gestures, forgetting that much of communication is about the social context, about who you are communicating with,” Stolk said.

The word “bank,” for example, would be interpreted one way if you’re holding a credit card but a different way if you’re holding a fishing pole. Without context, making a “V” with two fingers could mean victory, the number two, or “these are the two fingers I broke.”

“All these subtleties are quite crucial to understanding one another,” Stolk said, perhaps more so than the words and signals that computers and many neuroscientists focus on as the key to communication. “In fact, we can understand one another without language, without words and signs that already have a shared meaning.”

Will computers ever truly understand what we're saying?
A game in which players try to communicate the rules without talking or even seeing one another helps neuroscientists isolate the parts of the brain responsible for mutual understanding. Credit: Arjen Stolk, UC Berkeley

Babies and parents, not to mention strangers lacking a common language, communicate effectively all the time, based solely on gestures and a shared context they build up over even a short time.

Stolk argues that scientists and engineers should focus more on the contextual aspects of mutual understanding, basing his argument on experimental evidence from that humans achieve nonverbal mutual understanding using unique computational and neural mechanisms. Some of the studies Stolk has conducted suggest that a breakdown in mutual understanding is behind social disorders such as autism.

“This shift in understanding how people communicate without any need for language provides a new theoretical and empirical foundation for understanding normal social communication, and provides a new window into understanding and treating disorders of social communication in neurological and neurodevelopmental disorders,” said Dr. Robert Knight, a UC Berkeley professor of psychology in the campus’s Helen Wills Neuroscience Institute and a professor of neurology and neurosurgery at UCSF.

Stolk and his colleagues discuss the importance of conceptual alignment for mutual understanding in an opinion piece appearing Jan. 11 in the journal Trends in Cognitive Sciences.

Brain scans pinpoint site for ‘meeting of minds’

To explore how brains achieve mutual understanding, Stolk created a game that requires two players to communicate the rules to each other solely by game movements, without talking or even seeing one another, eliminating the influence of language or gesture. He then placed both players in an fMRI (functional magnetic resonance imager) and scanned their brains as they nonverbally communicated with one another via computer.

He found that the same regions of the brain – located in the poorly understood right temporal lobe, just above the ear – became active in both players during attempts to communicate the rules of the game. Critically, the of the right temporal lobe maintained a steady, baseline activity throughout the game but became more active when one player suddenly understood what the other player was trying to communicate. The brain’s right hemisphere is more involved in abstract thought and social interactions than the left hemisphere.

“These regions in the right temporal lobe increase in activity the moment you establish a shared meaning for something, but not when you communicate a signal,” Stolk said. “The better the players got at understanding each other, the more active this region became.”

This means that both players are building a similar conceptual framework in the same area of the brain, constantly testing one another to make sure their concepts align, and updating only when new information changes that mutual understanding. The results were reported in 2014 in the Proceedings of the National Academy of Sciences.

“It is surprising,” said Stolk, “that for both the communicator, who has static input while she is planning her move, and the addressee, who is observing dynamic visual input during the game, the same region of the brain becomes more active over the course of the experiment as they improve their mutual understanding.”

Robots’ statistical reasoning

Robots and computers, on the other hand, converse based on a statistical analysis of a word’s meaning, Stolk said. If you usually use the word “bank” to mean a place to cash a check, then that will be the assumed meaning in a conversation, even when the conversation is about fishing.

“Apple’s Siri focuses on statistical regularities, but communication is not about statistical regularities,” he said. “Statistical regularities may get you far, but it is not how the brain does it. In order for computers to communicate with us, they would need a cognitive architecture that continuously captures and updates the conceptual space shared with their communication partner during a conversation.”

Hypothetically, such a dynamic conceptual framework would allow computers to resolve the intrinsically ambiguous communication signals produced by a real person, including drawing upon information stored years earlier.

Stolk’s studies have pinpointed other brain areas critical to mutual understanding. In a 2014 study, he used brain stimulation to disrupt a rear portion of the temporal lobe and found that it is important for integrating incoming signals with knowledge from previous interactions. A later study found that in patients with damage to the frontal lobe (the ventromedial prefrontal cortex), decisions to communicate are no longer fine-tuned to stored knowledge about an addressee. Both studies could explain why such patients appear socially awkward in everyday social interactions.

Stolk plans future studies with Knight using fine-tuned brain mapping on the actual surfaces of the brains of volunteers, so-called electrocorticography.

Stolk said he wrote the new paper in hopes of moving the study of communication to a new level with a focus on conceptual alignment.

“Most cognitive neuroscientists focus on the signals themselves, on the words, gestures and their statistical relationships, ignoring the underlying conceptual ability that we use during communication and the flexibility of everyday life,” he said. “Language is very helpful, but it is a tool for communication, it is not communication per se. By focusing on language, you may be focusing on the tool, not on the underlying mechanism, the cognitive architecture we have in our that helps us to communicate.”