ArsTechnica

Ars TechnicaArs Technica UKRegisterLog in

Home
Main Menu
My Stories: 9
Forums
Subscribe
Jobs
LIVE: Microsoft Windows 10 hardware
GEAR & GADGETS / PRODUCT NEWS & REVIEWS
Android 6.0 Marshmallow, thoroughly reviewed
Marshmallow brings a lot of user-requested features but still has no update solu
tion.
by Ron Amadeo - Oct 5, 2015 11:06pm WIB
Share Tweet
183
Google Now on Tap
Ron Amadeo
A Now on Tap query for an article and the keywords it seems to have latched onto
. The subject of the article, Bitcoin, is last in the results.
Google Now on Tap is the latest feature in the "Googleization" of Android. Now o
n Tap performs a Google Search or brings up a Google Now card by using the conte
xt of the current screen. Just hold down on the home button and a card will pop
up related to the current screen. If the main screen displays a YouTube video of
a famous actor, you'll get more information about that person.
To make this work, holding down the home button allows Google to "read" the scre
en by combing through the Android text fields and view hierarchy, sending Google
that text plus a screenshot. This will work not only in Google apps and in the
browser but in third-party apps too, as long as they use the standard Android fr
amework bits. Apps can be data providers for the feature, too; for instance, in
the famous actor example above, the IMDB app could provide a link into the app t
o display the actor's filmography.
The interface of Now on Tap is a little misleading, which was one of our biggest
problems with it at first. The design tempts you to tap on the top section of a
card it's the biggest touch area on the screen, with big, bold text and usually a
n image. But the top section of a card merely performs a Google Search, which is
often the least useful action you can take. Now on Tap shows a glimmer of usefu
lness when you retrain yourself to hunt through the tiny app icons at the bottom
of a card. Here app indexing allows you to do things like jump directly from a
song to the music video on YouTube, check out a place mentioned in e-mail by hop
ping directly to Yelp, or go directly to a famous person's Twitter account. Pick
the right icon and it's like you've skipped the search engine and that's a genuin
e time saver.
When you open Google Now on Tap, you're usually looking to do something very, ve
ry specific. We would open it looking for a link to make a calendar event or for
information about a specific item or person, but Now on Tap would often whiff w
hen it came to finding the exact information we wanted. When you type something
into Google Search, Google knows with 100 percent accuracy what the subject of y
our search is it's the words you typed into the box. When you know the subject, yo
u can guess a lot about what the user wants to do. If the subject is a place, th
e user probably wants a map, operating hours, or a phone number. If the subject
is a person, the user probably want to see that person's Wikipedia entry, Twitte
r account, or news about them.

But Google Now on Tap doesn't know the subject of your search it has to guess from
reading the current page. That adds a whole layer of complexity to a search eng
ine, and it's often the source of our complaints with the feature. When you open
Now on Tap, it feels like you're asking Google to read your mind; too often, "N
ope, you did not correctly read my mind" is the outcome.
Remember those old Bing "Search Overload" commercials where people would rattle
off search results for random spoken words? Now on Tap feels a bit like that. It
pulls in cards for random nouns on the page and you can sift through them but with
seemingly no "weight" given to search results, this can often be a crapshoot. Fo
r instance, the first card in Now on Tap for every one of my e-mails is "Ars Tec
hnica," because it's part of my e-mail address. But of course, what's important
to me is usually the person's name at the end of the e-mail, and any times, date
s, or locations in the body. For our Bitcoin article (this article loaded in Chr
ome), the Bitcoin card was last. The word "Bitcoin" was in the article six times
, and it's the only search term in the title! It should be first.
Sometimes the feature returns nothing at all. In Google Hangouts, when we opened
Now on Tap in response to the question "How is the weather now?" we got... noth
ing. We almost think it would be better to have some kind of overlay pop up wher
e you can highlight the thing on the screen you want to search for.
Now on Tap isn't just about pop-up cards. The "OK Google" voice commands will no
w use information from the current screen, too. This allows you to ask contextua
l questions like, "How far is this from here?" when looking at a place in Yelp o
r Foursquare. Google is (usually) smart enough to figure out that "this" means t
he location currently displayed in the app. When looking at an address in an e-m
ail, we could just say "navigate there" and Google Maps started right up and sta
rted navigating. When it works, it's a great use of context.
Other times, though, Now on Tap doesn't use enough context. We asked "how old is
he" when watching a Stephen Colbert video and were told "73" the age of Stephen H
awking.
One of the downsides to Google Now on Tap is that the opening gesture long pressin
g the home screen used to belong to Google Now. There's no way to quickly open Goo
gle Now from any screen anymore. You have to go to the home screen and swipe lef
t. The gesture also once allowed you to have a third-party home screen and still
easily access Google Now, but you can't do that anymore, either. One option is
to disable Now on Tap completely, and then the old Google Now gesture will work.
Like Google's Search engine, Now On Tap should continually get better, and it's
still early days for the new interface. Algorithms will be tweaked, apps will be
tter support it, and Google is free to update it at any time since it's just par
t of the Google app. At the moment, though, it's hard to "trust" the feature to
do something useful. You're either pleasantly surprised when it works or disappo
inted when it doesn't. Even the most positive of these outcomes, "surprise," isn
't really what we're looking for in a computer interface. "Reliable" and "predic
table" are qualities that are more up our alley, and Now on Tap isn't any of tho
se things yet. We often try it while thinking "I wonder what this will do," but
so far it's been hard to integrate it into any kind of workflow.
The Assist API and Google's continuing commitment to user choice
Enlarge / The assist API settings.
Ron Amadeo
Google Now on Tap is made possible by the "Assist API," a plugin system that sen
ds designated "Assistant" app information about what the user is currently looki

ng at. This data is described in AssistStructure.ViewNode and AssistStructure.Wi

ndowNode. Among other things, it includes all the text in the current view, not
just what is currently on-screen. The assistant app only gets data about the cur
rent view if the user long-presses on the home button it's not a passive scanner.
Still though, if anyone builds an alternative assistant app, we'd imagine users
will want to be very careful about which app they choose as an assistant.
If all of this sounds like a privacy nightmare, the assistant feature can be tur
ned off in the settings. Head to Settings -> Apps -> Configure Apps (the gear bu
tton) -> Default Apps -> Assist and Voice Input and turn off everything. Here, u
sers can also set which app has access to the Assist API (there can only be one)
and pick between sending the app text-only or text and a screenshot.
Sending the assistant app your bank data would be pretty dumb, so developers hav
e a way to opt out of the assist API. Setting the "FLAG_SECURE" flag will stop d
ata from going to the assistant app when the user long presses on the home butto
n. The problem with opting out on a developer-by-developer basis is that your da
ta security is up to the developer. It would be nice to let users take initiativ
e and block the assistant API from certain apps on their own. A user's security
and privacy should never rely on an Android app developer being quick to update
their app. Put the user in control.
Developers can also send extra data to the assistant if they want by using the A
ssistContent class. While Google Now on Tap will take all the on-screen data and
try to figure out the context on its own, the AssistContent class lets develope
rs do a lot of this work. It packages up data to hand to the Assistant app at th
e user's request. We imagine that this would be good for apps that don't follow
the standard Android framework layouts Now on Tap uses to determine context but
still want to play nice with the feature.
With as much flack as Google gets from the FTC and other regulatory bodies for r
estricting rivals' access to Android, the Assist API shows Google's commitment t
o user choice. While developing Google Now on Tap, Google could have used a bunc
h of obfuscated, undocumented APIs that lock the operating system into its propr
ietary solution. Instead, it developed an API alongside Google Now on Tap.
Google added a bunch of hooks to AOSP and then made the most of those hooks, but
any company is free to use them the way Google did. Users can also change from
Google's solution to any other assistant app in the settings. So if Microsoft ch
ooses to make a Bing or Cortana Assistant for Android Marshmallow (or expand its
current Bing Snapshots functionality), the company is free to do so and users a
re free to choose it.
Every part of Android
hird-party app can do
but any app can plug
ext-to-speech engine,

is like this. Google takes over the home screen, but any t
that. Google is the system-wide, always-on voice provider,
into the hotword API. You can replace Google's keyboard, t
SMS app, browser, or phone app.

Granted, Google certainly gets a home field advantage. The platform APIs are dev
eloped alongside Google's proprietary apps, so they are created to do everything
the app team needs. The APIs are also usually developed in secret, giving Googl
e a head start. And of course, Google gets to be the default. Despite all of tha
t, Marshmallow offers more third-party compatibility and user choice than any ot
her mobile OS can boast.
PAGE: 1 2 3 4 5 ... 11 12 NEXT ?
READER COMMENTS 183
Share
615
Tweet

526
Google
1264
Reddit
1523
Ron Amadeo / Ron is the Reviews Editor at Ars Technica, where he specializes in
Android OS and Google products. He is always on the hunt for a new gadget and lo
ves to rip things apart to see how they work.
? OLDER STORY NEWER STORY ?
YOU MAY ALSO LIKE
Google posts Android 6.0 Marshmallow system images for Nexus devices
Paranoid Android redux:

going dark with Silent Circle s Blackphone 2

Marshmallow updates start rolling out to older Nexus devices next week
Liveblog: Google s 2015 Nexus launch event
Huawei s first Android Wear watch is a beautiful yet basic timepiece
Android Pay Google s second take on mobile payments rolls out today
A billion Android phones are vulnerable to new Stagefright bugs
SITE LINKS
About Us
Advertise with us
Contact Us
Reprints
SUBSCRIPTIONS
Subscribe to Ars
MORE READING
RSS Feeds
Newsletters
Visit Ars Technica UK
CONDE NAST SITES
Reddit
Wired
Vanity Fair
Style
Details
VIEW MOBILE SITE
2015 Cond Nast. All rights reserved
Use of this Site constitutes acceptance of our User Agreement (effective 1/2/14)
and Privacy Policy (effective 1/2/14), and Ars Technica Addendum (effective 5/1
7/2012)
Your California Privacy Rights
The material on this site may not be reproduced, distributed, transmitted, cache
d or otherwise used, except with the prior written permission of Cond Nast.
Ad ChoicesArsTechnica
Ars TechnicaArs Technica UKRegisterLog in
Home
Main Menu
My Stories: 9
Forums

Subscribe
Jobs
LIVE: Microsoft Windows 10 hardware
GEAR & GADGETS / PRODUCT NEWS & REVIEWS
Android 6.0 Marshmallow, thoroughly reviewed
Marshmallow brings a lot of user-requested features but still has no update solu
tion.
by Ron Amadeo - Oct 5, 2015 11:06pm WIB
Share Tweet
183
Google Now on Tap
Ron Amadeo
A Now on Tap query for an article and the keywords it seems to have latched onto
. The subject of the article, Bitcoin, is last in the results.
Google Now on Tap is the latest feature in the "Googleization" of Android. Now o
n Tap performs a Google Search or brings up a Google Now card by using the conte
xt of the current screen. Just hold down on the home button and a card will pop
up related to the current screen. If the main screen displays a YouTube video of
a famous actor, you'll get more information about that person.
To make this work, holding down the home button allows Google to "read" the scre
en by combing through the Android text fields and view hierarchy, sending Google
that text plus a screenshot. This will work not only in Google apps and in the
browser but in third-party apps too, as long as they use the standard Android fr
amework bits. Apps can be data providers for the feature, too; for instance, in
the famous actor example above, the IMDB app could provide a link into the app t
o display the actor's filmography.
The interface of Now on Tap is a little misleading, which was one of our biggest
problems with it at first. The design tempts you to tap on the top section of a
card it's the biggest touch area on the screen, with big, bold text and usually a
n image. But the top section of a card merely performs a Google Search, which is
often the least useful action you can take. Now on Tap shows a glimmer of usefu
lness when you retrain yourself to hunt through the tiny app icons at the bottom
of a card. Here app indexing allows you to do things like jump directly from a
song to the music video on YouTube, check out a place mentioned in e-mail by hop
ping directly to Yelp, or go directly to a famous person's Twitter account. Pick
the right icon and it's like you've skipped the search engine and that's a genuin
e time saver.
When you open Google Now on Tap, you're usually looking to do something very, ve
ry specific. We would open it looking for a link to make a calendar event or for
information about a specific item or person, but Now on Tap would often whiff w
hen it came to finding the exact information we wanted. When you type something
into Google Search, Google knows with 100 percent accuracy what the subject of y
our search is it's the words you typed into the box. When you know the subject, yo
u can guess a lot about what the user wants to do. If the subject is a place, th
e user probably wants a map, operating hours, or a phone number. If the subject
is a person, the user probably want to see that person's Wikipedia entry, Twitte
r account, or news about them.
But Google Now on Tap doesn't know the subject of your search it has to guess from
reading the current page. That adds a whole layer of complexity to a search eng
ine, and it's often the source of our complaints with the feature. When you open
Now on Tap, it feels like you're asking Google to read your mind; too often, "N
ope, you did not correctly read my mind" is the outcome.

Remember those old Bing "Search Overload" commercials where people would rattle
off search results for random spoken words? Now on Tap feels a bit like that. It
pulls in cards for random nouns on the page and you can sift through them but with
seemingly no "weight" given to search results, this can often be a crapshoot. Fo
r instance, the first card in Now on Tap for every one of my e-mails is "Ars Tec
hnica," because it's part of my e-mail address. But of course, what's important
to me is usually the person's name at the end of the e-mail, and any times, date
s, or locations in the body. For our Bitcoin article (this article loaded in Chr
ome), the Bitcoin card was last. The word "Bitcoin" was in the article six times
, and it's the only search term in the title! It should be first.
Sometimes the feature returns nothing at all. In Google Hangouts, when we opened
Now on Tap in response to the question "How is the weather now?" we got... noth
ing. We almost think it would be better to have some kind of overlay pop up wher
e you can highlight the thing on the screen you want to search for.
Now on Tap isn't just about pop-up cards. The "OK Google" voice commands will no
w use information from the current screen, too. This allows you to ask contextua
l questions like, "How far is this from here?" when looking at a place in Yelp o
r Foursquare. Google is (usually) smart enough to figure out that "this" means t
he location currently displayed in the app. When looking at an address in an e-m
ail, we could just say "navigate there" and Google Maps started right up and sta
rted navigating. When it works, it's a great use of context.
Other times, though, Now on Tap doesn't use enough context. We asked "how old is
he" when watching a Stephen Colbert video and were told "73" the age of Stephen H
awking.
One of the downsides to Google Now on Tap is that the opening gesture long pressin
g the home screen used to belong to Google Now. There's no way to quickly open Goo
gle Now from any screen anymore. You have to go to the home screen and swipe lef
t. The gesture also once allowed you to have a third-party home screen and still
easily access Google Now, but you can't do that anymore, either. One option is
to disable Now on Tap completely, and then the old Google Now gesture will work.
Like Google's Search engine, Now On Tap should continually get better, and it's
still early days for the new interface. Algorithms will be tweaked, apps will be
tter support it, and Google is free to update it at any time since it's just par
t of the Google app. At the moment, though, it's hard to "trust" the feature to
do something useful. You're either pleasantly surprised when it works or disappo
inted when it doesn't. Even the most positive of these outcomes, "surprise," isn
't really what we're looking for in a computer interface. "Reliable" and "predic
table" are qualities that are more up our alley, and Now on Tap isn't any of tho
se things yet. We often try it while thinking "I wonder what this will do," but
so far it's been hard to integrate it into any kind of workflow.
The Assist API and Google's continuing commitment to user choice
Enlarge / The assist API settings.
Ron Amadeo
Google Now on Tap is made possible by the "Assist API," a plugin system that sen
ds designated "Assistant" app information about what the user is currently looki
ng at. This data is described in AssistStructure.ViewNode and AssistStructure.Wi
ndowNode. Among other things, it includes all the text in the current view, not
just what is currently on-screen. The assistant app only gets data about the cur
rent view if the user long-presses on the home button it's not a passive scanner.
Still though, if anyone builds an alternative assistant app, we'd imagine users
will want to be very careful about which app they choose as an assistant.

If all of this sounds like a privacy nightmare, the assistant feature can be tur
ned off in the settings. Head to Settings -> Apps -> Configure Apps (the gear bu
tton) -> Default Apps -> Assist and Voice Input and turn off everything. Here, u
sers can also set which app has access to the Assist API (there can only be one)
and pick between sending the app text-only or text and a screenshot.
Sending the assistant app your bank data would be pretty dumb, so developers hav
e a way to opt out of the assist API. Setting the "FLAG_SECURE" flag will stop d
ata from going to the assistant app when the user long presses on the home butto
n. The problem with opting out on a developer-by-developer basis is that your da
ta security is up to the developer. It would be nice to let users take initiativ
e and block the assistant API from certain apps on their own. A user's security
and privacy should never rely on an Android app developer being quick to update
their app. Put the user in control.
Developers can also send extra data to the assistant if they want by using the A
ssistContent class. While Google Now on Tap will take all the on-screen data and
try to figure out the context on its own, the AssistContent class lets develope
rs do a lot of this work. It packages up data to hand to the Assistant app at th
e user's request. We imagine that this would be good for apps that don't follow
the standard Android framework layouts Now on Tap uses to determine context but
still want to play nice with the feature.
With as much flack as Google gets from the FTC and other regulatory bodies for r
estricting rivals' access to Android, the Assist API shows Google's commitment t
o user choice. While developing Google Now on Tap, Google could have used a bunc
h of obfuscated, undocumented APIs that lock the operating system into its propr
ietary solution. Instead, it developed an API alongside Google Now on Tap.
Google added a bunch of hooks to AOSP and then made the most of those hooks, but
any company is free to use them the way Google did. Users can also change from
Google's solution to any other assistant app in the settings. So if Microsoft ch
ooses to make a Bing or Cortana Assistant for Android Marshmallow (or expand its
current Bing Snapshots functionality), the company is free to do so and users a
re free to choose it.
Every part of Android
hird-party app can do
but any app can plug
ext-to-speech engine,

is like this. Google takes over the home screen, but any t
that. Google is the system-wide, always-on voice provider,
into the hotword API. You can replace Google's keyboard, t
SMS app, browser, or phone app.

Granted, Google certainly gets a home field advantage. The platform APIs are dev
eloped alongside Google's proprietary apps, so they are created to do everything
the app team needs. The APIs are also usually developed in secret, giving Googl
e a head start. And of course, Google gets to be the default. Despite all of tha
t, Marshmallow offers more third-party compatibility and user choice than any ot
her mobile OS can boast.
PAGE: 1 2 3 4 5 ... 11 12 NEXT ?
READER COMMENTS 183
Share
615
Tweet
526
Google
1264
Reddit
1523

Ron Amadeo / Ron is the Reviews Editor at Ars Technica, where he specializes in
Android OS and Google products. He is always on the hunt for a new gadget and lo
ves to rip things apart to see how they work.
? OLDER STORY NEWER STORY ?
YOU MAY ALSO LIKE
Google posts Android 6.0 Marshmallow system images for Nexus devices
Paranoid Android redux:

going dark with Silent Circle s Blackphone 2

Marshmallow updates start rolling out to older Nexus devices next week
Liveblog: Google s 2015 Nexus launch event
Huawei s first Android Wear watch is a beautiful yet basic timepiece
Android Pay Google s second take on mobile payments rolls out today
A billion Android phones are vulnerable to new Stagefright bugs
SITE LINKS
About Us
Advertise with us
Contact Us
Reprints
SUBSCRIPTIONS
Subscribe to Ars
MORE READING
RSS Feeds
Newsletters
Visit Ars Technica UK
CONDE NAST SITES
Reddit
Wired
Vanity Fair
Style
Details
VIEW MOBILE SITE
2015 Cond Nast. All rights reserved
Use of this Site constitutes acceptance of our User Agreement (effective 1/2/14)
and Privacy Policy (effective 1/2/14), and Ars Technica Addendum (effective 5/1
7/2012)
Your California Privacy Rights
The material on this site may not be reproduced, distributed, transmitted, cache
d or otherwise used, except with the prior written permission of Cond Nast.
Ad ChoicesArsTechnica
Ars TechnicaArs Technica UKRegisterLog in
Home
Main Menu
My Stories: 9
Forums
Subscribe
Jobs
LIVE: Microsoft Windows 10 hardware
GEAR & GADGETS / PRODUCT NEWS & REVIEWS
Android 6.0 Marshmallow, thoroughly reviewed

Marshmallow brings a lot of user-requested features but still has no update solu
tion.
by Ron Amadeo - Oct 5, 2015 11:06pm WIB
Share Tweet
183
Google Now on Tap
Ron Amadeo
A Now on Tap query for an article and the keywords it seems to have latched onto
. The subject of the article, Bitcoin, is last in the results.
Google Now on Tap is the latest feature in the "Googleization" of Android. Now o
n Tap performs a Google Search or brings up a Google Now card by using the conte
xt of the current screen. Just hold down on the home button and a card will pop
up related to the current screen. If the main screen displays a YouTube video of
a famous actor, you'll get more information about that person.
To make this work, holding down the home button allows Google to "read" the scre
en by combing through the Android text fields and view hierarchy, sending Google
that text plus a screenshot. This will work not only in Google apps and in the
browser but in third-party apps too, as long as they use the standard Android fr
amework bits. Apps can be data providers for the feature, too; for instance, in
the famous actor example above, the IMDB app could provide a link into the app t
o display the actor's filmography.
The interface of Now on Tap is a little misleading, which was one of our biggest
problems with it at first. The design tempts you to tap on the top section of a
card it's the biggest touch area on the screen, with big, bold text and usually a
n image. But the top section of a card merely performs a Google Search, which is
often the least useful action you can take. Now on Tap shows a glimmer of usefu
lness when you retrain yourself to hunt through the tiny app icons at the bottom
of a card. Here app indexing allows you to do things like jump directly from a
song to the music video on YouTube, check out a place mentioned in e-mail by hop
ping directly to Yelp, or go directly to a famous person's Twitter account. Pick
the right icon and it's like you've skipped the search engine and that's a genuin
e time saver.
When you open Google Now on Tap, you're usually looking to do something very, ve
ry specific. We would open it looking for a link to make a calendar event or for
information about a specific item or person, but Now on Tap would often whiff w
hen it came to finding the exact information we wanted. When you type something
into Google Search, Google knows with 100 percent accuracy what the subject of y
our search is it's the words you typed into the box. When you know the subject, yo
u can guess a lot about what the user wants to do. If the subject is a place, th
e user probably wants a map, operating hours, or a phone number. If the subject
is a person, the user probably want to see that person's Wikipedia entry, Twitte
r account, or news about them.
But Google Now on Tap doesn't know the subject of your search it has to guess from
reading the current page. That adds a whole layer of complexity to a search eng
ine, and it's often the source of our complaints with the feature. When you open
Now on Tap, it feels like you're asking Google to read your mind; too often, "N
ope, you did not correctly read my mind" is the outcome.
Remember those old Bing "Search Overload" commercials where people would rattle
off search results for random spoken words? Now on Tap feels a bit like that. It
pulls in cards for random nouns on the page and you can sift through them but with
seemingly no "weight" given to search results, this can often be a crapshoot. Fo
r instance, the first card in Now on Tap for every one of my e-mails is "Ars Tec

hnica," because it's part of my e-mail address. But of course, what's important
to me is usually the person's name at the end of the e-mail, and any times, date
s, or locations in the body. For our Bitcoin article (this article loaded in Chr
ome), the Bitcoin card was last. The word "Bitcoin" was in the article six times
, and it's the only search term in the title! It should be first.
Sometimes the feature returns nothing at all. In Google Hangouts, when we opened
Now on Tap in response to the question "How is the weather now?" we got... noth
ing. We almost think it would be better to have some kind of overlay pop up wher
e you can highlight the thing on the screen you want to search for.
Now on Tap isn't just about pop-up cards. The "OK Google" voice commands will no
w use information from the current screen, too. This allows you to ask contextua
l questions like, "How far is this from here?" when looking at a place in Yelp o
r Foursquare. Google is (usually) smart enough to figure out that "this" means t
he location currently displayed in the app. When looking at an address in an e-m
ail, we could just say "navigate there" and Google Maps started right up and sta
rted navigating. When it works, it's a great use of context.
Other times, though, Now on Tap doesn't use enough context. We asked "how old is
he" when watching a Stephen Colbert video and were told "73" the age of Stephen H
awking.
One of the downsides to Google Now on Tap is that the opening gesture long pressin
g the home screen used to belong to Google Now. There's no way to quickly open Goo
gle Now from any screen anymore. You have to go to the home screen and swipe lef
t. The gesture also once allowed you to have a third-party home screen and still
easily access Google Now, but you can't do that anymore, either. One option is
to disable Now on Tap completely, and then the old Google Now gesture will work.
Like Google's Search engine, Now On Tap should continually get better, and it's
still early days for the new interface. Algorithms will be tweaked, apps will be
tter support it, and Google is free to update it at any time since it's just par
t of the Google app. At the moment, though, it's hard to "trust" the feature to
do something useful. You're either pleasantly surprised when it works or disappo
inted when it doesn't. Even the most positive of these outcomes, "surprise," isn
't really what we're looking for in a computer interface. "Reliable" and "predic
table" are qualities that are more up our alley, and Now on Tap isn't any of tho
se things yet. We often try it while thinking "I wonder what this will do," but
so far it's been hard to integrate it into any kind of workflow.
The Assist API and Google's continuing commitment to user choice
Enlarge / The assist API settings.
Ron Amadeo
Google Now on Tap is made possible by the "Assist API," a plugin system that sen
ds designated "Assistant" app information about what the user is currently looki
ng at. This data is described in AssistStructure.ViewNode and AssistStructure.Wi
ndowNode. Among other things, it includes all the text in the current view, not
just what is currently on-screen. The assistant app only gets data about the cur
rent view if the user long-presses on the home button it's not a passive scanner.
Still though, if anyone builds an alternative assistant app, we'd imagine users
will want to be very careful about which app they choose as an assistant.
If all of this sounds like a privacy nightmare, the assistant feature can be tur
ned off in the settings. Head to Settings -> Apps -> Configure Apps (the gear bu
tton) -> Default Apps -> Assist and Voice Input and turn off everything. Here, u
sers can also set which app has access to the Assist API (there can only be one)
and pick between sending the app text-only or text and a screenshot.

Sending the assistant app your bank data would be pretty dumb, so developers hav
e a way to opt out of the assist API. Setting the "FLAG_SECURE" flag will stop d
ata from going to the assistant app when the user long presses on the home butto
n. The problem with opting out on a developer-by-developer basis is that your da
ta security is up to the developer. It would be nice to let users take initiativ
e and block the assistant API from certain apps on their own. A user's security
and privacy should never rely on an Android app developer being quick to update
their app. Put the user in control.
Developers can also send extra data to the assistant if they want by using the A
ssistContent class. While Google Now on Tap will take all the on-screen data and
try to figure out the context on its own, the AssistContent class lets develope
rs do a lot of this work. It packages up data to hand to the Assistant app at th
e user's request. We imagine that this would be good for apps that don't follow
the standard Android framework layouts Now on Tap uses to determine context but
still want to play nice with the feature.
With as much flack as Google gets from the FTC and other regulatory bodies for r
estricting rivals' access to Android, the Assist API shows Google's commitment t
o user choice. While developing Google Now on Tap, Google could have used a bunc
h of obfuscated, undocumented APIs that lock the operating system into its propr
ietary solution. Instead, it developed an API alongside Google Now on Tap.
Google added a bunch of hooks to AOSP and then made the most of those hooks, but
any company is free to use them the way Google did. Users can also change from
Google's solution to any other assistant app in the settings. So if Microsoft ch
ooses to make a Bing or Cortana Assistant for Android Marshmallow (or expand its
current Bing Snapshots functionality), the company is free to do so and users a
re free to choose it.
Every part of Android
hird-party app can do
but any app can plug
ext-to-speech engine,

is like this. Google takes over the home screen, but any t
that. Google is the system-wide, always-on voice provider,
into the hotword API. You can replace Google's keyboard, t
SMS app, browser, or phone app.

Granted, Google certainly gets a home field advantage. The platform APIs are dev
eloped alongside Google's proprietary apps, so they are created to do everything
the app team needs. The APIs are also usually developed in secret, giving Googl
e a head start. And of course, Google gets to be the default. Despite all of tha
t, Marshmallow offers more third-party compatibility and user choice than any ot
her mobile OS can boast.
PAGE: 1 2 3 4 5 ... 11 12 NEXT ?
READER COMMENTS 183
Share
615
Tweet
526
Google
1264
Reddit
1523
Ron Amadeo / Ron is the Reviews Editor at Ars Technica, where he specializes in
Android OS and Google products. He is always on the hunt for a new gadget and lo
ves to rip things apart to see how they work.
? OLDER STORY NEWER STORY ?
YOU MAY ALSO LIKE

Google posts Android 6.0 Marshmallow system images for Nexus devices
Paranoid Android redux:

going dark with Silent Circle s Blackphone 2

Marshmallow updates start rolling out to older Nexus devices next week
Liveblog: Google s 2015 Nexus launch event
Huawei s first Android Wear watch is a beautiful yet basic timepiece
Android Pay Google s second take on mobile payments rolls out today
A billion Android phones are vulnerable to new Stagefright bugs
SITE LINKS
About Us
Advertise with us
Contact Us
Reprints
SUBSCRIPTIONS
Subscribe to Ars
MORE READING
RSS Feeds
Newsletters
Visit Ars Technica UK
CONDE NAST SITES
Reddit
Wired
Vanity Fair
Style
Details
VIEW MOBILE SITE
2015 Cond Nast. All rights reserved
Use of this Site constitutes acceptance of our User Agreement (effective 1/2/14)
and Privacy Policy (effective 1/2/14), and Ars Technica Addendum (effective 5/1
7/2012)
Your California Privacy Rights
The material on this site may not be reproduced, distributed, transmitted, cache
d or otherwise used, except with the prior written permission of Cond Nast.
Ad ChoicesArsTechnica
Ars TechnicaArs Technica UKRegisterLog in
Home
Main Menu
My Stories: 9
Forums
Subscribe
Jobs
LIVE: Microsoft Windows 10 hardware
GEAR & GADGETS / PRODUCT NEWS & REVIEWS
Android 6.0 Marshmallow, thoroughly reviewed
Marshmallow brings a lot of user-requested features but still has no update solu
tion.
by Ron Amadeo - Oct 5, 2015 11:06pm WIB
Share Tweet
183

Google Now on Tap

Ron Amadeo
A Now on Tap query for an article and the keywords it seems to have latched onto
. The subject of the article, Bitcoin, is last in the results.
Google Now on Tap is the latest feature in the "Googleization" of Android. Now o
n Tap performs a Google Search or brings up a Google Now card by using the conte
xt of the current screen. Just hold down on the home button and a card will pop
up related to the current screen. If the main screen displays a YouTube video of
a famous actor, you'll get more information about that person.
To make this work, holding down the home button allows Google to "read" the scre
en by combing through the Android text fields and view hierarchy, sending Google
that text plus a screenshot. This will work not only in Google apps and in the
browser but in third-party apps too, as long as they use the standard Android fr
amework bits. Apps can be data providers for the feature, too; for instance, in
the famous actor example above, the IMDB app could provide a link into the app t
o display the actor's filmography.
The interface of Now on Tap is a little misleading, which was one of our biggest
problems with it at first. The design tempts you to tap on the top section of a
card it's the biggest touch area on the screen, with big, bold text and usually a
n image. But the top section of a card merely performs a Google Search, which is
often the least useful action you can take. Now on Tap shows a glimmer of usefu
lness when you retrain yourself to hunt through the tiny app icons at the bottom
of a card. Here app indexing allows you to do things like jump directly from a
song to the music video on YouTube, check out a place mentioned in e-mail by hop
ping directly to Yelp, or go directly to a famous person's Twitter account. Pick
the right icon and it's like you've skipped the search engine and that's a genuin
e time saver.
When you open Google Now on Tap, you're usually looking to do something very, ve
ry specific. We would open it looking for a link to make a calendar event or for
information about a specific item or person, but Now on Tap would often whiff w
hen it came to finding the exact information we wanted. When you type something
into Google Search, Google knows with 100 percent accuracy what the subject of y
our search is it's the words you typed into the box. When you know the subject, yo
u can guess a lot about what the user wants to do. If the subject is a place, th
e user probably wants a map, operating hours, or a phone number. If the subject
is a person, the user probably want to see that person's Wikipedia entry, Twitte
r account, or news about them.
But Google Now on Tap doesn't know the subject of your search it has to guess from
reading the current page. That adds a whole layer of complexity to a search eng
ine, and it's often the source of our complaints with the feature. When you open
Now on Tap, it feels like you're asking Google to read your mind; too often, "N
ope, you did not correctly read my mind" is the outcome.
Remember those old Bing "Search Overload" commercials where people would rattle
off search results for random spoken words? Now on Tap feels a bit like that. It
pulls in cards for random nouns on the page and you can sift through them but with
seemingly no "weight" given to search results, this can often be a crapshoot. Fo
r instance, the first card in Now on Tap for every one of my e-mails is "Ars Tec
hnica," because it's part of my e-mail address. But of course, what's important
to me is usually the person's name at the end of the e-mail, and any times, date
s, or locations in the body. For our Bitcoin article (this article loaded in Chr
ome), the Bitcoin card was last. The word "Bitcoin" was in the article six times
, and it's the only search term in the title! It should be first.

Sometimes the feature returns nothing at all. In Google Hangouts, when we opened
Now on Tap in response to the question "How is the weather now?" we got... noth
ing. We almost think it would be better to have some kind of overlay pop up wher
e you can highlight the thing on the screen you want to search for.
Now on Tap isn't just about pop-up cards. The "OK Google" voice commands will no
w use information from the current screen, too. This allows you to ask contextua
l questions like, "How far is this from here?" when looking at a place in Yelp o
r Foursquare. Google is (usually) smart enough to figure out that "this" means t
he location currently displayed in the app. When looking at an address in an e-m
ail, we could just say "navigate there" and Google Maps started right up and sta
rted navigating. When it works, it's a great use of context.
Other times, though, Now on Tap doesn't use enough context. We asked "how old is
he" when watching a Stephen Colbert video and were told "73" the age of Stephen H
awking.
One of the downsides to Google Now on Tap is that the opening gesture long pressin
g the home screen used to belong to Google Now. There's no way to quickly open Goo
gle Now from any screen anymore. You have to go to the home screen and swipe lef
t. The gesture also once allowed you to have a third-party home screen and still
easily access Google Now, but you can't do that anymore, either. One option is
to disable Now on Tap completely, and then the old Google Now gesture will work.
Like Google's Search engine, Now On Tap should continually get better, and it's
still early days for the new interface. Algorithms will be tweaked, apps will be
tter support it, and Google is free to update it at any time since it's just par
t of the Google app. At the moment, though, it's hard to "trust" the feature to
do something useful. You're either pleasantly surprised when it works or disappo
inted when it doesn't. Even the most positive of these outcomes, "surprise," isn
't really what we're looking for in a computer interface. "Reliable" and "predic
table" are qualities that are more up our alley, and Now on Tap isn't any of tho
se things yet. We often try it while thinking "I wonder what this will do," but
so far it's been hard to integrate it into any kind of workflow.
The Assist API and Google's continuing commitment to user choice
Enlarge / The assist API settings.
Ron Amadeo
Google Now on Tap is made possible by the "Assist API," a plugin system that sen
ds designated "Assistant" app information about what the user is currently looki
ng at. This data is described in AssistStructure.ViewNode and AssistStructure.Wi
ndowNode. Among other things, it includes all the text in the current view, not
just what is currently on-screen. The assistant app only gets data about the cur
rent view if the user long-presses on the home button it's not a passive scanner.
Still though, if anyone builds an alternative assistant app, we'd imagine users
will want to be very careful about which app they choose as an assistant.
If all of this sounds like a privacy nightmare, the assistant feature can be tur
ned off in the settings. Head to Settings -> Apps -> Configure Apps (the gear bu
tton) -> Default Apps -> Assist and Voice Input and turn off everything. Here, u
sers can also set which app has access to the Assist API (there can only be one)
and pick between sending the app text-only or text and a screenshot.
Sending the assistant app your bank data would be pretty dumb, so developers hav
e a way to opt out of the assist API. Setting the "FLAG_SECURE" flag will stop d
ata from going to the assistant app when the user long presses on the home butto
n. The problem with opting out on a developer-by-developer basis is that your da
ta security is up to the developer. It would be nice to let users take initiativ

e and block the assistant API from certain apps on their own. A user's security
and privacy should never rely on an Android app developer being quick to update
their app. Put the user in control.
Developers can also send extra data to the assistant if they want by using the A
ssistContent class. While Google Now on Tap will take all the on-screen data and
try to figure out the context on its own, the AssistContent class lets develope
rs do a lot of this work. It packages up data to hand to the Assistant app at th
e user's request. We imagine that this would be good for apps that don't follow
the standard Android framework layouts Now on Tap uses to determine context but
still want to play nice with the feature.
With as much flack as Google gets from the FTC and other regulatory bodies for r
estricting rivals' access to Android, the Assist API shows Google's commitment t
o user choice. While developing Google Now on Tap, Google could have used a bunc
h of obfuscated, undocumented APIs that lock the operating system into its propr
ietary solution. Instead, it developed an API alongside Google Now on Tap.
Google added a bunch of hooks to AOSP and then made the most of those hooks, but
any company is free to use them the way Google did. Users can also change from
Google's solution to any other assistant app in the settings. So if Microsoft ch
ooses to make a Bing or Cortana Assistant for Android Marshmallow (or expand its
current Bing Snapshots functionality), the company is free to do so and users a
re free to choose it.
Every part of Android
hird-party app can do
but any app can plug
ext-to-speech engine,

is like this. Google takes over the home screen, but any t
that. Google is the system-wide, always-on voice provider,
into the hotword API. You can replace Google's keyboard, t
SMS app, browser, or phone app.

Granted, Google certainly gets a home field advantage. The platform APIs are dev
eloped alongside Google's proprietary apps, so they are created to do everything
the app team needs. The APIs are also usually developed in secret, giving Googl
e a head start. And of course, Google gets to be the default. Despite all of tha
t, Marshmallow offers more third-party compatibility and user choice than any ot
her mobile OS can boast.
PAGE: 1 2 3 4 5 ... 11 12 NEXT ?
READER COMMENTS 183
Share
615
Tweet
526
Google
1264
Reddit
1523
Ron Amadeo / Ron is the Reviews Editor at Ars Technica, where he specializes in
Android OS and Google products. He is always on the hunt for a new gadget and lo
ves to rip things apart to see how they work.
? OLDER STORY NEWER STORY ?
YOU MAY ALSO LIKE
Google posts Android 6.0 Marshmallow system images for Nexus devices
Paranoid Android redux:

going dark with Silent Circle s Blackphone 2

Marshmallow updates start rolling out to older Nexus devices next week

Liveblog: Google s 2015 Nexus launch event

Huawei s first Android Wear watch is a beautiful yet basic timepiece
Android Pay Google s second take on mobile payments rolls out today
A billion Android phones are vulnerable to new Stagefright bugs
SITE LINKS
About Us
Advertise with us
Contact Us
Reprints
SUBSCRIPTIONS
Subscribe to Ars
MORE READING
RSS Feeds
Newsletters
Visit Ars Technica UK
CONDE NAST SITES
Reddit
Wired
Vanity Fair
Style
Details
VIEW MOBILE SITE
2015 Cond Nast. All rights reserved
Use of this Site constitutes acceptance of our User Agreement (effective 1/2/14)
and Privacy Policy (effective 1/2/14), and Ars Technica Addendum (effective 5/1
7/2012)
Your California Privacy Rights
The material on this site may not be reproduced, distributed, transmitted, cache
d or otherwise used, except with the prior written permission of Cond Nast.
Ad ChoicesArsTechnica
Ars TechnicaArs Technica UKRegisterLog in
Home
Main Menu
My Stories: 9
Forums
Subscribe
Jobs
LIVE: Microsoft Windows 10 hardware
GEAR & GADGETS / PRODUCT NEWS & REVIEWS
Android 6.0 Marshmallow, thoroughly reviewed
Marshmallow brings a lot of user-requested features but still has no update solu
tion.
by Ron Amadeo - Oct 5, 2015 11:06pm WIB
Share Tweet
183
Google Now on Tap
Ron Amadeo
A Now on Tap query for an article and the keywords it seems to have latched onto
. The subject of the article, Bitcoin, is last in the results.

Google Now on Tap is the latest feature in the "Googleization" of Android. Now o
n Tap performs a Google Search or brings up a Google Now card by using the conte
xt of the current screen. Just hold down on the home button and a card will pop
up related to the current screen. If the main screen displays a YouTube video of
a famous actor, you'll get more information about that person.
To make this work, holding down the home button allows Google to "read" the scre
en by combing through the Android text fields and view hierarchy, sending Google
that text plus a screenshot. This will work not only in Google apps and in the
browser but in third-party apps too, as long as they use the standard Android fr
amework bits. Apps can be data providers for the feature, too; for instance, in
the famous actor example above, the IMDB app could provide a link into the app t
o display the actor's filmography.
The interface of Now on Tap is a little misleading, which was one of our biggest
problems with it at first. The design tempts you to tap on the top section of a
card it's the biggest touch area on the screen, with big, bold text and usually a
n image. But the top section of a card merely performs a Google Search, which is
often the least useful action you can take. Now on Tap shows a glimmer of usefu
lness when you retrain yourself to hunt through the tiny app icons at the bottom
of a card. Here app indexing allows you to do things like jump directly from a
song to the music video on YouTube, check out a place mentioned in e-mail by hop
ping directly to Yelp, or go directly to a famous person's Twitter account. Pick
the right icon and it's like you've skipped the search engine and that's a genuin
e time saver.
When you open Google Now on Tap, you're usually looking to do something very, ve
ry specific. We would open it looking for a link to make a calendar event or for
information about a specific item or person, but Now on Tap would often whiff w
hen it came to finding the exact information we wanted. When you type something
into Google Search, Google knows with 100 percent accuracy what the subject of y
our search is it's the words you typed into the box. When you know the subject, yo
u can guess a lot about what the user wants to do. If the subject is a place, th
e user probably wants a map, operating hours, or a phone number. If the subject
is a person, the user probably want to see that person's Wikipedia entry, Twitte
r account, or news about them.
But Google Now on Tap doesn't know the subject of your search it has to guess from
reading the current page. That adds a whole layer of complexity to a search eng
ine, and it's often the source of our complaints with the feature. When you open
Now on Tap, it feels like you're asking Google to read your mind; too often, "N
ope, you did not correctly read my mind" is the outcome.
Remember those old Bing "Search Overload" commercials where people would rattle
off search results for random spoken words? Now on Tap feels a bit like that. It
pulls in cards for random nouns on the page and you can sift through them but with
seemingly no "weight" given to search results, this can often be a crapshoot. Fo
r instance, the first card in Now on Tap for every one of my e-mails is "Ars Tec
hnica," because it's part of my e-mail address. But of course, what's important
to me is usually the person's name at the end of the e-mail, and any times, date
s, or locations in the body. For our Bitcoin article (this article loaded in Chr
ome), the Bitcoin card was last. The word "Bitcoin" was in the article six times
, and it's the only search term in the title! It should be first.
Sometimes the feature returns nothing at all. In Google Hangouts, when we opened
Now on Tap in response to the question "How is the weather now?" we got... noth
ing. We almost think it would be better to have some kind of overlay pop up wher
e you can highlight the thing on the screen you want to search for.
Now on Tap isn't just about pop-up cards. The "OK Google" voice commands will no

w use information from the current screen, too. This allows you to ask contextua
l questions like, "How far is this from here?" when looking at a place in Yelp o
r Foursquare. Google is (usually) smart enough to figure out that "this" means t
he location currently displayed in the app. When looking at an address in an e-m
ail, we could just say "navigate there" and Google Maps started right up and sta
rted navigating. When it works, it's a great use of context.
Other times, though, Now on Tap doesn't use enough context. We asked "how old is
he" when watching a Stephen Colbert video and were told "73" the age of Stephen H
awking.
One of the downsides to Google Now on Tap is that the opening gesture long pressin
g the home screen used to belong to Google Now. There's no way to quickly open Goo
gle Now from any screen anymore. You have to go to the home screen and swipe lef
t. The gesture also once allowed you to have a third-party home screen and still
easily access Google Now, but you can't do that anymore, either. One option is
to disable Now on Tap completely, and then the old Google Now gesture will work.
Like Google's Search engine, Now On Tap should continually get better, and it's
still early days for the new interface. Algorithms will be tweaked, apps will be
tter support it, and Google is free to update it at any time since it's just par
t of the Google app. At the moment, though, it's hard to "trust" the feature to
do something useful. You're either pleasantly surprised when it works or disappo
inted when it doesn't. Even the most positive of these outcomes, "surprise," isn
't really what we're looking for in a computer interface. "Reliable" and "predic
table" are qualities that are more up our alley, and Now on Tap isn't any of tho
se things yet. We often try it while thinking "I wonder what this will do," but
so far it's been hard to integrate it into any kind of workflow.
The Assist API and Google's continuing commitment to user choice
Enlarge / The assist API settings.
Ron Amadeo
Google Now on Tap is made possible by the "Assist API," a plugin system that sen
ds designated "Assistant" app information about what the user is currently looki
ng at. This data is described in AssistStructure.ViewNode and AssistStructure.Wi
ndowNode. Among other things, it includes all the text in the current view, not
just what is currently on-screen. The assistant app only gets data about the cur
rent view if the user long-presses on the home button it's not a passive scanner.
Still though, if anyone builds an alternative assistant app, we'd imagine users
will want to be very careful about which app they choose as an assistant.
If all of this sounds like a privacy nightmare, the assistant feature can be tur
ned off in the settings. Head to Settings -> Apps -> Configure Apps (the gear bu
tton) -> Default Apps -> Assist and Voice Input and turn off everything. Here, u
sers can also set which app has access to the Assist API (there can only be one)
and pick between sending the app text-only or text and a screenshot.
Sending the assistant app your bank data would be pretty dumb, so developers hav
e a way to opt out of the assist API. Setting the "FLAG_SECURE" flag will stop d
ata from going to the assistant app when the user long presses on the home butto
n. The problem with opting out on a developer-by-developer basis is that your da
ta security is up to the developer. It would be nice to let users take initiativ
e and block the assistant API from certain apps on their own. A user's security
and privacy should never rely on an Android app developer being quick to update
their app. Put the user in control.
Developers can also send extra data to the assistant if they want by using the A
ssistContent class. While Google Now on Tap will take all the on-screen data and

try to figure out the context on its own, the AssistContent class lets develope
rs do a lot of this work. It packages up data to hand to the Assistant app at th
e user's request. We imagine that this would be good for apps that don't follow
the standard Android framework layouts Now on Tap uses to determine context but
still want to play nice with the feature.
With as much flack as Google gets from the FTC and other regulatory bodies for r
estricting rivals' access to Android, the Assist API shows Google's commitment t
o user choice. While developing Google Now on Tap, Google could have used a bunc
h of obfuscated, undocumented APIs that lock the operating system into its propr
ietary solution. Instead, it developed an API alongside Google Now on Tap.
Google added a bunch of hooks to AOSP and then made the most of those hooks, but
any company is free to use them the way Google did. Users can also change from
Google's solution to any other assistant app in the settings. So if Microsoft ch
ooses to make a Bing or Cortana Assistant for Android Marshmallow (or expand its
current Bing Snapshots functionality), the company is free to do so and users a
re free to choose it.
Every part of Android
hird-party app can do
but any app can plug
ext-to-speech engine,

is like this. Google takes over the home screen, but any t
that. Google is the system-wide, always-on voice provider,
into the hotword API. You can replace Google's keyboard, t
SMS app, browser, or phone app.

Granted, Google certainly gets a home field advantage. The platform APIs are dev
eloped alongside Google's proprietary apps, so they are created to do everything
the app team needs. The APIs are also usually developed in secret, giving Googl
e a head start. And of course, Google gets to be the default. Despite all of tha
t, Marshmallow offers more third-party compatibility and user choice than any ot
her mobile OS can boast.
PAGE: 1 2 3 4 5 ... 11 12 NEXT ?
READER COMMENTS 183
Share
615
Tweet
526
Google
1264
Reddit
1523
Ron Amadeo / Ron is the Reviews Editor at Ars Technica, where he specializes in
Android OS and Google products. He is always on the hunt for a new gadget and lo
ves to rip things apart to see how they work.
? OLDER STORY NEWER STORY ?
YOU MAY ALSO LIKE
Google posts Android 6.0 Marshmallow system images for Nexus devices
Paranoid Android redux:

going dark with Silent Circle s Blackphone 2

Marshmallow updates start rolling out to older Nexus devices next week
Liveblog: Google s 2015 Nexus launch event
Huawei s first Android Wear watch is a beautiful yet basic timepiece
Android Pay Google s second take on mobile payments rolls out today

A billion Android phones are vulnerable to new Stagefright bugs

SITE LINKS
About Us
Advertise with us
Contact Us
Reprints
SUBSCRIPTIONS
Subscribe to Ars
MORE READING
RSS Feeds
Newsletters
Visit Ars Technica UK
CONDE NAST SITES
Reddit
Wired
Vanity Fair
Style
Details
VIEW MOBILE SITE
2015 Cond Nast. All rights reserved
Use of this Site constitutes acceptance of our User Agreement (effective 1/2/14)
and Privacy Policy (effective 1/2/14), and Ars Technica Addendum (effective 5/1
7/2012)
Your California Privacy Rights
The material on this site may not be reproduced, distributed, transmitted, cache
d or otherwise used, except with the prior written permission of Cond Nast.
Ad ChoicesArsTechnica
Ars TechnicaArs Technica UKRegisterLog in
Home
Main Menu
My Stories: 9
Forums
Subscribe
Jobs
LIVE: Microsoft Windows 10 hardware
GEAR & GADGETS / PRODUCT NEWS & REVIEWS
Android 6.0 Marshmallow, thoroughly reviewed
Marshmallow brings a lot of user-requested features but still has no update solu
tion.
by Ron Amadeo - Oct 5, 2015 11:06pm WIB
Share Tweet
183
Google Now on Tap
Ron Amadeo
A Now on Tap query for an article and the keywords it seems to have latched onto
. The subject of the article, Bitcoin, is last in the results.
Google Now on Tap is the latest feature in the "Googleization" of Android. Now o
n Tap performs a Google Search or brings up a Google Now card by using the conte
xt of the current screen. Just hold down on the home button and a card will pop
up related to the current screen. If the main screen displays a YouTube video of
a famous actor, you'll get more information about that person.

To make this work, holding down the home button allows Google to "read" the scre
en by combing through the Android text fields and view hierarchy, sending Google
that text plus a screenshot. This will work not only in Google apps and in the
browser but in third-party apps too, as long as they use the standard Android fr
amework bits. Apps can be data providers for the feature, too; for instance, in
the famous actor example above, the IMDB app could provide a link into the app t
o display the actor's filmography.
The interface of Now on Tap is a little misleading, which was one of our biggest
problems with it at first. The design tempts you to tap on the top section of a
card it's the biggest touch area on the screen, with big, bold text and usually a
n image. But the top section of a card merely performs a Google Search, which is
often the least useful action you can take. Now on Tap shows a glimmer of usefu
lness when you retrain yourself to hunt through the tiny app icons at the bottom
of a card. Here app indexing allows you to do things like jump directly from a
song to the music video on YouTube, check out a place mentioned in e-mail by hop
ping directly to Yelp, or go directly to a famous person's Twitter account. Pick
the right icon and it's like you've skipped the search engine and that's a genuin
e time saver.
When you open Google Now on Tap, you're usually looking to do something very, ve
ry specific. We would open it looking for a link to make a calendar event or for
information about a specific item or person, but Now on Tap would often whiff w
hen it came to finding the exact information we wanted. When you type something
into Google Search, Google knows with 100 percent accuracy what the subject of y
our search is it's the words you typed into the box. When you know the subject, yo
u can guess a lot about what the user wants to do. If the subject is a place, th
e user probably wants a map, operating hours, or a phone number. If the subject
is a person, the user probably want to see that person's Wikipedia entry, Twitte
r account, or news about them.
But Google Now on Tap doesn't know the subject of your search it has to guess from
reading the current page. That adds a whole layer of complexity to a search eng
ine, and it's often the source of our complaints with the feature. When you open
Now on Tap, it feels like you're asking Google to read your mind; too often, "N
ope, you did not correctly read my mind" is the outcome.
Remember those old Bing "Search Overload" commercials where people would rattle
off search results for random spoken words? Now on Tap feels a bit like that. It
pulls in cards for random nouns on the page and you can sift through them but with
seemingly no "weight" given to search results, this can often be a crapshoot. Fo
r instance, the first card in Now on Tap for every one of my e-mails is "Ars Tec
hnica," because it's part of my e-mail address. But of course, what's important
to me is usually the person's name at the end of the e-mail, and any times, date
s, or locations in the body. For our Bitcoin article (this article loaded in Chr
ome), the Bitcoin card was last. The word "Bitcoin" was in the article six times
, and it's the only search term in the title! It should be first.
Sometimes the feature returns nothing at all. In Google Hangouts, when we opened
Now on Tap in response to the question "How is the weather now?" we got... noth
ing. We almost think it would be better to have some kind of overlay pop up wher
e you can highlight the thing on the screen you want to search for.
Now on Tap isn't just about pop-up cards. The "OK Google" voice commands will no
w use information from the current screen, too. This allows you to ask contextua
l questions like, "How far is this from here?" when looking at a place in Yelp o
r Foursquare. Google is (usually) smart enough to figure out that "this" means t
he location currently displayed in the app. When looking at an address in an e-m
ail, we could just say "navigate there" and Google Maps started right up and sta
rted navigating. When it works, it's a great use of context.

Other times, though, Now on Tap doesn't use enough context. We asked "how old is
he" when watching a Stephen Colbert video and were told "73" the age of Stephen H
awking.
One of the downsides to Google Now on Tap is that the opening gesture long pressin
g the home screen used to belong to Google Now. There's no way to quickly open Goo
gle Now from any screen anymore. You have to go to the home screen and swipe lef
t. The gesture also once allowed you to have a third-party home screen and still
easily access Google Now, but you can't do that anymore, either. One option is
to disable Now on Tap completely, and then the old Google Now gesture will work.
Like Google's Search engine, Now On Tap should continually get better, and it's
still early days for the new interface. Algorithms will be tweaked, apps will be
tter support it, and Google is free to update it at any time since it's just par
t of the Google app. At the moment, though, it's hard to "trust" the feature to
do something useful. You're either pleasantly surprised when it works or disappo
inted when it doesn't. Even the most positive of these outcomes, "surprise," isn
't really what we're looking for in a computer interface. "Reliable" and "predic
table" are qualities that are more up our alley, and Now on Tap isn't any of tho
se things yet. We often try it while thinking "I wonder what this will do," but
so far it's been hard to integrate it into any kind of workflow.
The Assist API and Google's continuing commitment to user choice
Enlarge / The assist API settings.
Ron Amadeo
Google Now on Tap is made possible by the "Assist API," a plugin system that sen
ds designated "Assistant" app information about what the user is currently looki
ng at. This data is described in AssistStructure.ViewNode and AssistStructure.Wi
ndowNode. Among other things, it includes all the text in the current view, not
just what is currently on-screen. The assistant app only gets data about the cur
rent view if the user long-presses on the home button it's not a passive scanner.
Still though, if anyone builds an alternative assistant app, we'd imagine users
will want to be very careful about which app they choose as an assistant.
If all of this sounds like a privacy nightmare, the assistant feature can be tur
ned off in the settings. Head to Settings -> Apps -> Configure Apps (the gear bu
tton) -> Default Apps -> Assist and Voice Input and turn off everything. Here, u
sers can also set which app has access to the Assist API (there can only be one)
and pick between sending the app text-only or text and a screenshot.
Sending the assistant app your bank data would be pretty dumb, so developers hav
e a way to opt out of the assist API. Setting the "FLAG_SECURE" flag will stop d
ata from going to the assistant app when the user long presses on the home butto
n. The problem with opting out on a developer-by-developer basis is that your da
ta security is up to the developer. It would be nice to let users take initiativ
e and block the assistant API from certain apps on their own. A user's security
and privacy should never rely on an Android app developer being quick to update
their app. Put the user in control.
Developers can also send extra data to the assistant if they want by using the A
ssistContent class. While Google Now on Tap will take all the on-screen data and
try to figure out the context on its own, the AssistContent class lets develope
rs do a lot of this work. It packages up data to hand to the Assistant app at th
e user's request. We imagine that this would be good for apps that don't follow
the standard Android framework layouts Now on Tap uses to determine context but
still want to play nice with the feature.

With as much flack as Google gets from the FTC and other regulatory bodies for r
estricting rivals' access to Android, the Assist API shows Google's commitment t
o user choice. While developing Google Now on Tap, Google could have used a bunc
h of obfuscated, undocumented APIs that lock the operating system into its propr
ietary solution. Instead, it developed an API alongside Google Now on Tap.
Google added a bunch of hooks to AOSP and then made the most of those hooks, but
any company is free to use them the way Google did. Users can also change from
Google's solution to any other assistant app in the settings. So if Microsoft ch
ooses to make a Bing or Cortana Assistant for Android Marshmallow (or expand its
current Bing Snapshots functionality), the company is free to do so and users a
re free to choose it.
Every part of Android
hird-party app can do
but any app can plug
ext-to-speech engine,

is like this. Google takes over the home screen, but any t
that. Google is the system-wide, always-on voice provider,
into the hotword API. You can replace Google's keyboard, t
SMS app, browser, or phone app.

Granted, Google certainly gets a home field advantage. The platform APIs are dev
eloped alongside Google's proprietary apps, so they are created to do everything
the app team needs. The APIs are also usually developed in secret, giving Googl
e a head start. And of course, Google gets to be the default. Despite all of tha
t, Marshmallow offers more third-party compatibility and user choice than any ot
her mobile OS can boast.
PAGE: 1 2 3 4 5 ... 11 12 NEXT ?
READER COMMENTS 183
Share
615
Tweet
526
Google
1264
Reddit
1523
Ron Amadeo / Ron is the Reviews Editor at Ars Technica, where he specializes in
Android OS and Google products. He is always on the hunt for a new gadget and lo
ves to rip things apart to see how they work.
? OLDER STORY NEWER STORY ?
YOU MAY ALSO LIKE
Google posts Android 6.0 Marshmallow system images for Nexus devices
Paranoid Android redux:

going dark with Silent Circle s Blackphone 2

Marshmallow updates start rolling out to older Nexus devices next week
Liveblog: Google s 2015 Nexus launch event
Huawei s first Android Wear watch is a beautiful yet basic timepiece
Android Pay Google s second take on mobile payments rolls out today
A billion Android phones are vulnerable to new Stagefright bugs
SITE LINKS
About Us
Advertise with us
Contact Us
Reprints

SUBSCRIPTIONS
Subscribe to Ars
MORE READING
RSS Feeds
Newsletters
Visit Ars Technica UK
CONDE NAST SITES
Reddit
Wired
Vanity Fair
Style
Details
VIEW MOBILE SITE
2015 Cond Nast. All rights reserved
Use of this Site constitutes acceptance of our User Agreement (effective 1/2/14)
and Privacy Policy (effective 1/2/14), and Ars Technica Addendum (effective 5/1
7/2012)
Your California Privacy Rights
The material on this site may not be reproduced, distributed, transmitted, cache
d or otherwise used, except with the prior written permission of Cond Nast.
Ad ChoicesArsTechnica
Ars TechnicaArs Technica UKRegisterLog in
Home
Main Menu
My Stories: 9
Forums
Subscribe
Jobs
LIVE: Microsoft Windows 10 hardware
GEAR & GADGETS / PRODUCT NEWS & REVIEWS
Android 6.0 Marshmallow, thoroughly reviewed
Marshmallow brings a lot of user-requested features but still has no update solu
tion.
by Ron Amadeo - Oct 5, 2015 11:06pm WIB
Share Tweet
183
Google Now on Tap
Ron Amadeo
A Now on Tap query for an article and the keywords it seems to have latched onto
. The subject of the article, Bitcoin, is last in the results.
Google Now on Tap is the latest feature in the "Googleization" of Android. Now o
n Tap performs a Google Search or brings up a Google Now card by using the conte
xt of the current screen. Just hold down on the home button and a card will pop
up related to the current screen. If the main screen displays a YouTube video of
a famous actor, you'll get more information about that person.
To make this work, holding down the home button allows Google to "read" the scre
en by combing through the Android text fields and view hierarchy, sending Google
that text plus a screenshot. This will work not only in Google apps and in the
browser but in third-party apps too, as long as they use the standard Android fr
amework bits. Apps can be data providers for the feature, too; for instance, in
the famous actor example above, the IMDB app could provide a link into the app t

o display the actor's filmography.

The interface of Now on Tap is a little misleading, which was one of our biggest
problems with it at first. The design tempts you to tap on the top section of a
card it's the biggest touch area on the screen, with big, bold text and usually a
n image. But the top section of a card merely performs a Google Search, which is
often the least useful action you can take. Now on Tap shows a glimmer of usefu
lness when you retrain yourself to hunt through the tiny app icons at the bottom
of a card. Here app indexing allows you to do things like jump directly from a
song to the music video on YouTube, check out a place mentioned in e-mail by hop
ping directly to Yelp, or go directly to a famous person's Twitter account. Pick
the right icon and it's like you've skipped the search engine and that's a genuin
e time saver.
When you open Google Now on Tap, you're usually looking to do something very, ve
ry specific. We would open it looking for a link to make a calendar event or for
information about a specific item or person, but Now on Tap would often whiff w
hen it came to finding the exact information we wanted. When you type something
into Google Search, Google knows with 100 percent accuracy what the subject of y
our search is it's the words you typed into the box. When you know the subject, yo
u can guess a lot about what the user wants to do. If the subject is a place, th
e user probably wants a map, operating hours, or a phone number. If the subject
is a person, the user probably want to see that person's Wikipedia entry, Twitte
r account, or news about them.
But Google Now on Tap doesn't know the subject of your search it has to guess from
reading the current page. That adds a whole layer of complexity to a search eng
ine, and it's often the source of our complaints with the feature. When you open
Now on Tap, it feels like you're asking Google to read your mind; too often, "N
ope, you did not correctly read my mind" is the outcome.
Remember those old Bing "Search Overload" commercials where people would rattle
off search results for random spoken words? Now on Tap feels a bit like that. It
pulls in cards for random nouns on the page and you can sift through them but with
seemingly no "weight" given to search results, this can often be a crapshoot. Fo
r instance, the first card in Now on Tap for every one of my e-mails is "Ars Tec
hnica," because it's part of my e-mail address. But of course, what's important
to me is usually the person's name at the end of the e-mail, and any times, date
s, or locations in the body. For our Bitcoin article (this article loaded in Chr
ome), the Bitcoin card was last. The word "Bitcoin" was in the article six times
, and it's the only search term in the title! It should be first.
Sometimes the feature returns nothing at all. In Google Hangouts, when we opened
Now on Tap in response to the question "How is the weather now?" we got... noth
ing. We almost think it would be better to have some kind of overlay pop up wher
e you can highlight the thing on the screen you want to search for.
Now on Tap isn't just about pop-up cards. The "OK Google" voice commands will no
w use information from the current screen, too. This allows you to ask contextua
l questions like, "How far is this from here?" when looking at a place in Yelp o
r Foursquare. Google is (usually) smart enough to figure out that "this" means t
he location currently displayed in the app. When looking at an address in an e-m
ail, we could just say "navigate there" and Google Maps started right up and sta
rted navigating. When it works, it's a great use of context.
Other times, though, Now on Tap doesn't use enough context. We asked "how old is
he" when watching a Stephen Colbert video and were told "73" the age of Stephen H
awking.
One of the downsides to Google Now on Tap is that the opening gesture long pressin

g the home screen used to belong to Google Now. There's no way to quickly open Goo
gle Now from any screen anymore. You have to go to the home screen and swipe lef
t. The gesture also once allowed you to have a third-party home screen and still
easily access Google Now, but you can't do that anymore, either. One option is
to disable Now on Tap completely, and then the old Google Now gesture will work.
Like Google's Search engine, Now On Tap should continually get better, and it's
still early days for the new interface. Algorithms will be tweaked, apps will be
tter support it, and Google is free to update it at any time since it's just par
t of the Google app. At the moment, though, it's hard to "trust" the feature to
do something useful. You're either pleasantly surprised when it works or disappo
inted when it doesn't. Even the most positive of these outcomes, "surprise," isn
't really what we're looking for in a computer interface. "Reliable" and "predic
table" are qualities that are more up our alley, and Now on Tap isn't any of tho
se things yet. We often try it while thinking "I wonder what this will do," but
so far it's been hard to integrate it into any kind of workflow.
The Assist API and Google's continuing commitment to user choice
Enlarge / The assist API settings.
Ron Amadeo
Google Now on Tap is made possible by the "Assist API," a plugin system that sen
ds designated "Assistant" app information about what the user is currently looki
ng at. This data is described in AssistStructure.ViewNode and AssistStructure.Wi
ndowNode. Among other things, it includes all the text in the current view, not
just what is currently on-screen. The assistant app only gets data about the cur
rent view if the user long-presses on the home button it's not a passive scanner.
Still though, if anyone builds an alternative assistant app, we'd imagine users
will want to be very careful about which app they choose as an assistant.
If all of this sounds like a privacy nightmare, the assistant feature can be tur
ned off in the settings. Head to Settings -> Apps -> Configure Apps (the gear bu
tton) -> Default Apps -> Assist and Voice Input and turn off everything. Here, u
sers can also set which app has access to the Assist API (there can only be one)
and pick between sending the app text-only or text and a screenshot.
Sending the assistant app your bank data would be pretty dumb, so developers hav
e a way to opt out of the assist API. Setting the "FLAG_SECURE" flag will stop d
ata from going to the assistant app when the user long presses on the home butto
n. The problem with opting out on a developer-by-developer basis is that your da
ta security is up to the developer. It would be nice to let users take initiativ
e and block the assistant API from certain apps on their own. A user's security
and privacy should never rely on an Android app developer being quick to update
their app. Put the user in control.
Developers can also send extra data to the assistant if they want by using the A
ssistContent class. While Google Now on Tap will take all the on-screen data and
try to figure out the context on its own, the AssistContent class lets develope
rs do a lot of this work. It packages up data to hand to the Assistant app at th
e user's request. We imagine that this would be good for apps that don't follow
the standard Android framework layouts Now on Tap uses to determine context but
still want to play nice with the feature.
With as much flack as Google gets from the FTC and other regulatory bodies for r
estricting rivals' access to Android, the Assist API shows Google's commitment t
o user choice. While developing Google Now on Tap, Google could have used a bunc
h of obfuscated, undocumented APIs that lock the operating system into its propr
ietary solution. Instead, it developed an API alongside Google Now on Tap.

Google added a bunch of hooks to AOSP and then made the most of those hooks, but
any company is free to use them the way Google did. Users can also change from
Google's solution to any other assistant app in the settings. So if Microsoft ch
ooses to make a Bing or Cortana Assistant for Android Marshmallow (or expand its
current Bing Snapshots functionality), the company is free to do so and users a
re free to choose it.
Every part of Android
hird-party app can do
but any app can plug
ext-to-speech engine,

is like this. Google takes over the home screen, but any t
that. Google is the system-wide, always-on voice provider,
into the hotword API. You can replace Google's keyboard, t
SMS app, browser, or phone app.

Granted, Google certainly gets a home field advantage. The platform APIs are dev
eloped alongside Google's proprietary apps, so they are created to do everything
the app team needs. The APIs are also usually developed in secret, giving Googl
e a head start. And of course, Google gets to be the default. Despite all of tha
t, Marshmallow offers more third-party compatibility and user choice than any ot
her mobile OS can boast.
PAGE: 1 2 3 4 5 ... 11 12 NEXT ?
READER COMMENTS 183
Share
615
Tweet
526
Google
1264
Reddit
1523
Ron Amadeo / Ron is the Reviews Editor at Ars Technica, where he specializes in
Android OS and Google products. He is always on the hunt for a new gadget and lo
ves to rip things apart to see how they work.
? OLDER STORY NEWER STORY ?
YOU MAY ALSO LIKE
Google posts Android 6.0 Marshmallow system images for Nexus devices
Paranoid Android redux:

going dark with Silent Circle s Blackphone 2

Marshmallow updates start rolling out to older Nexus devices next week
Liveblog: Google s 2015 Nexus launch event
Huawei s first Android Wear watch is a beautiful yet basic timepiece
Android Pay Google s second take on mobile payments rolls out today
A billion Android phones are vulnerable to new Stagefright bugs
SITE LINKS
About Us
Advertise with us
Contact Us
Reprints
SUBSCRIPTIONS
Subscribe to Ars
MORE READING
RSS Feeds
Newsletters
Visit Ars Technica UK

CONDE NAST SITES

Reddit
Wired
Vanity Fair
Style
Details
VIEW MOBILE SITE
2015 Cond Nast. All rights reserved
Use of this Site constitutes acceptance of our User Agreement (effective 1/2/14)
and Privacy Policy (effective 1/2/14), and Ars Technica Addendum (effective 5/1
7/2012)
Your California Privacy Rights
The material on this site may not be reproduced, distributed, transmitted, cache
d or otherwise used, except with the prior written permission of Cond Nast.
Ad ChoicesArsTechnica
Ars TechnicaArs Technica UKRegisterLog in
Home
Main Menu
My Stories: 9
Forums
Subscribe
Jobs
LIVE: Microsoft Windows 10 hardware
GEAR & GADGETS / PRODUCT NEWS & REVIEWS
Android 6.0 Marshmallow, thoroughly reviewed
Marshmallow brings a lot of user-requested features but still has no update solu
tion.
by Ron Amadeo - Oct 5, 2015 11:06pm WIB
Share Tweet
183
Google Now on Tap
Ron Amadeo
A Now on Tap query for an article and the keywords it seems to have latched onto
. The subject of the article, Bitcoin, is last in the results.
Google Now on Tap is the latest feature in the "Googleization" of Android. Now o
n Tap performs a Google Search or brings up a Google Now card by using the conte
xt of the current screen. Just hold down on the home button and a card will pop
up related to the current screen. If the main screen displays a YouTube video of
a famous actor, you'll get more information about that person.
To make this work, holding down the home button allows Google to "read" the scre
en by combing through the Android text fields and view hierarchy, sending Google
that text plus a screenshot. This will work not only in Google apps and in the
browser but in third-party apps too, as long as they use the standard Android fr
amework bits. Apps can be data providers for the feature, too; for instance, in
the famous actor example above, the IMDB app could provide a link into the app t
o display the actor's filmography.
The interface of Now on Tap is a little misleading, which was one of our biggest
problems with it at first. The design tempts you to tap on the top section of a
card it's the biggest touch area on the screen, with big, bold text and usually a
n image. But the top section of a card merely performs a Google Search, which is

often the least useful action you can take. Now on Tap shows a glimmer of usefu
lness when you retrain yourself to hunt through the tiny app icons at the bottom
of a card. Here app indexing allows you to do things like jump directly from a
song to the music video on YouTube, check out a place mentioned in e-mail by hop
ping directly to Yelp, or go directly to a famous person's Twitter account. Pick
the right icon and it's like you've skipped the search engine and that's a genuin
e time saver.
When you open Google Now on Tap, you're usually looking to do something very, ve
ry specific. We would open it looking for a link to make a calendar event or for
information about a specific item or person, but Now on Tap would often whiff w
hen it came to finding the exact information we wanted. When you type something
into Google Search, Google knows with 100 percent accuracy what the subject of y
our search is it's the words you typed into the box. When you know the subject, yo
u can guess a lot about what the user wants to do. If the subject is a place, th
e user probably wants a map, operating hours, or a phone number. If the subject
is a person, the user probably want to see that person's Wikipedia entry, Twitte
r account, or news about them.
But Google Now on Tap doesn't know the subject of your search it has to guess from
reading the current page. That adds a whole layer of complexity to a search eng
ine, and it's often the source of our complaints with the feature. When you open
Now on Tap, it feels like you're asking Google to read your mind; too often, "N
ope, you did not correctly read my mind" is the outcome.
Remember those old Bing "Search Overload" commercials where people would rattle
off search results for random spoken words? Now on Tap feels a bit like that. It
pulls in cards for random nouns on the page and you can sift through them but with
seemingly no "weight" given to search results, this can often be a crapshoot. Fo
r instance, the first card in Now on Tap for every one of my e-mails is "Ars Tec
hnica," because it's part of my e-mail address. But of course, what's important
to me is usually the person's name at the end of the e-mail, and any times, date
s, or locations in the body. For our Bitcoin article (this article loaded in Chr
ome), the Bitcoin card was last. The word "Bitcoin" was in the article six times
, and it's the only search term in the title! It should be first.
Sometimes the feature returns nothing at all. In Google Hangouts, when we opened
Now on Tap in response to the question "How is the weather now?" we got... noth
ing. We almost think it would be better to have some kind of overlay pop up wher
e you can highlight the thing on the screen you want to search for.
Now on Tap isn't just about pop-up cards. The "OK Google" voice commands will no
w use information from the current screen, too. This allows you to ask contextua
l questions like, "How far is this from here?" when looking at a place in Yelp o
r Foursquare. Google is (usually) smart enough to figure out that "this" means t
he location currently displayed in the app. When looking at an address in an e-m
ail, we could just say "navigate there" and Google Maps started right up and sta
rted navigating. When it works, it's a great use of context.
Other times, though, Now on Tap doesn't use enough context. We asked "how old is
he" when watching a Stephen Colbert video and were told "73" the age of Stephen H
awking.
One of the downsides to Google Now on Tap is that the opening gesture long pressin
g the home screen used to belong to Google Now. There's no way to quickly open Goo
gle Now from any screen anymore. You have to go to the home screen and swipe lef
t. The gesture also once allowed you to have a third-party home screen and still
easily access Google Now, but you can't do that anymore, either. One option is
to disable Now on Tap completely, and then the old Google Now gesture will work.

Like Google's Search engine, Now On Tap should continually get better, and it's
still early days for the new interface. Algorithms will be tweaked, apps will be
tter support it, and Google is free to update it at any time since it's just par
t of the Google app. At the moment, though, it's hard to "trust" the feature to
do something useful. You're either pleasantly surprised when it works or disappo
inted when it doesn't. Even the most positive of these outcomes, "surprise," isn
't really what we're looking for in a computer interface. "Reliable" and "predic
table" are qualities that are more up our alley, and Now on Tap isn't any of tho
se things yet. We often try it while thinking "I wonder what this will do," but
so far it's been hard to integrate it into any kind of workflow.
The Assist API and Google's continuing commitment to user choice
Enlarge / The assist API settings.
Ron Amadeo
Google Now on Tap is made possible by the "Assist API," a plugin system that sen
ds designated "Assistant" app information about what the user is currently looki
ng at. This data is described in AssistStructure.ViewNode and AssistStructure.Wi
ndowNode. Among other things, it includes all the text in the current view, not
just what is currently on-screen. The assistant app only gets data about the cur
rent view if the user long-presses on the home button it's not a passive scanner.
Still though, if anyone builds an alternative assistant app, we'd imagine users
will want to be very careful about which app they choose as an assistant.
If all of this sounds like a privacy nightmare, the assistant feature can be tur
ned off in the settings. Head to Settings -> Apps -> Configure Apps (the gear bu
tton) -> Default Apps -> Assist and Voice Input and turn off everything. Here, u
sers can also set which app has access to the Assist API (there can only be one)
and pick between sending the app text-only or text and a screenshot.
Sending the assistant app your bank data would be pretty dumb, so developers hav
e a way to opt out of the assist API. Setting the "FLAG_SECURE" flag will stop d
ata from going to the assistant app when the user long presses on the home butto
n. The problem with opting out on a developer-by-developer basis is that your da
ta security is up to the developer. It would be nice to let users take initiativ
e and block the assistant API from certain apps on their own. A user's security
and privacy should never rely on an Android app developer being quick to update
their app. Put the user in control.
Developers can also send extra data to the assistant if they want by using the A
ssistContent class. While Google Now on Tap will take all the on-screen data and
try to figure out the context on its own, the AssistContent class lets develope
rs do a lot of this work. It packages up data to hand to the Assistant app at th
e user's request. We imagine that this would be good for apps that don't follow
the standard Android framework layouts Now on Tap uses to determine context but
still want to play nice with the feature.
With as much flack as Google gets from the FTC and other regulatory bodies for r
estricting rivals' access to Android, the Assist API shows Google's commitment t
o user choice. While developing Google Now on Tap, Google could have used a bunc
h of obfuscated, undocumented APIs that lock the operating system into its propr
ietary solution. Instead, it developed an API alongside Google Now on Tap.
Google added a bunch of hooks to AOSP and then made the most of those hooks, but
any company is free to use them the way Google did. Users can also change from
Google's solution to any other assistant app in the settings. So if Microsoft ch
ooses to make a Bing or Cortana Assistant for Android Marshmallow (or expand its
current Bing Snapshots functionality), the company is free to do so and users a
re free to choose it.

Every part of Android

hird-party app can do
but any app can plug
ext-to-speech engine,

is like this. Google takes over the home screen, but any t
that. Google is the system-wide, always-on voice provider,
into the hotword API. You can replace Google's keyboard, t
SMS app, browser, or phone app.

Granted, Google certainly gets a home field advantage. The platform APIs are dev
eloped alongside Google's proprietary apps, so they are created to do everything
the app team needs. The APIs are also usually developed in secret, giving Googl
e a head start. And of course, Google gets to be the default. Despite all of tha
t, Marshmallow offers more third-party compatibility and user choice than any ot
her mobile OS can boast.
PAGE: 1 2 3 4 5 ... 11 12 NEXT ?
READER COMMENTS 183
Share
615
Tweet
526
Google
1264
Reddit
1523
Ron Amadeo / Ron is the Reviews Editor at Ars Technica, where he specializes in
Android OS and Google products. He is always on the hunt for a new gadget and lo
ves to rip things apart to see how they work.
? OLDER STORY NEWER STORY ?
YOU MAY ALSO LIKE
Google posts Android 6.0 Marshmallow system images for Nexus devices
Paranoid Android redux:

going dark with Silent Circle s Blackphone 2

Marshmallow updates start rolling out to older Nexus devices next week
Liveblog: Google s 2015 Nexus launch event
Huawei s first Android Wear watch is a beautiful yet basic timepiece
Android Pay Google s second take on mobile payments rolls out today
A billion Android phones are vulnerable to new Stagefright bugs
SITE LINKS
About Us
Advertise with us
Contact Us
Reprints
SUBSCRIPTIONS
Subscribe to Ars
MORE READING
RSS Feeds
Newsletters
Visit Ars Technica UK
CONDE NAST SITES
Reddit
Wired
Vanity Fair
Style
Details

VIEW MOBILE SITE

2015 Cond Nast. All rights reserved
Use of this Site constitutes acceptance of our User Agreement (effective 1/2/14)
and Privacy Policy (effective 1/2/14), and Ars Technica Addendum (effective 5/1
7/2012)
Your California Privacy Rights
The material on this site may not be reproduced, distributed, transmitted, cache
d or otherwise used, except with the prior written permission of Cond Nast.
Ad ChoicesArsTechnica
Ars TechnicaArs Technica UKRegisterLog in
Home
Main Menu
My Stories: 9
Forums
Subscribe
Jobs
LIVE: Microsoft Windows 10 hardware
GEAR & GADGETS / PRODUCT NEWS & REVIEWS
Android 6.0 Marshmallow, thoroughly reviewed
Marshmallow brings a lot of user-requested features but still has no update solu
tion.
by Ron Amadeo - Oct 5, 2015 11:06pm WIB
Share Tweet
183
Google Now on Tap
Ron Amadeo
A Now on Tap query for an article and the keywords it seems to have latched onto
. The subject of the article, Bitcoin, is last in the results.
Google Now on Tap is the latest feature in the "Googleization" of Android. Now o
n Tap performs a Google Search or brings up a Google Now card by using the conte
xt of the current screen. Just hold down on the home button and a card will pop
up related to the current screen. If the main screen displays a YouTube video of
a famous actor, you'll get more information about that person.
To make this work, holding down the home button allows Google to "read" the scre
en by combing through the Android text fields and view hierarchy, sending Google
that text plus a screenshot. This will work not only in Google apps and in the
browser but in third-party apps too, as long as they use the standard Android fr
amework bits. Apps can be data providers for the feature, too; for instance, in
the famous actor example above, the IMDB app could provide a link into the app t
o display the actor's filmography.
The interface of Now on Tap is a little misleading, which was one of our biggest
problems with it at first. The design tempts you to tap on the top section of a
card it's the biggest touch area on the screen, with big, bold text and usually a
n image. But the top section of a card merely performs a Google Search, which is
often the least useful action you can take. Now on Tap shows a glimmer of usefu
lness when you retrain yourself to hunt through the tiny app icons at the bottom
of a card. Here app indexing allows you to do things like jump directly from a
song to the music video on YouTube, check out a place mentioned in e-mail by hop
ping directly to Yelp, or go directly to a famous person's Twitter account. Pick
the right icon and it's like you've skipped the search engine and that's a genuin

e time saver.
When you open Google Now on Tap, you're usually looking to do something very, ve
ry specific. We would open it looking for a link to make a calendar event or for
information about a specific item or person, but Now on Tap would often whiff w
hen it came to finding the exact information we wanted. When you type something
into Google Search, Google knows with 100 percent accuracy what the subject of y
our search is it's the words you typed into the box. When you know the subject, yo
u can guess a lot about what the user wants to do. If the subject is a place, th
e user probably wants a map, operating hours, or a phone number. If the subject
is a person, the user probably want to see that person's Wikipedia entry, Twitte
r account, or news about them.
But Google Now on Tap doesn't know the subject of your search it has to guess from
reading the current page. That adds a whole layer of complexity to a search eng
ine, and it's often the source of our complaints with the feature. When you open
Now on Tap, it feels like you're asking Google to read your mind; too often, "N
ope, you did not correctly read my mind" is the outcome.
Remember those old Bing "Search Overload" commercials where people would rattle
off search results for random spoken words? Now on Tap feels a bit like that. It
pulls in cards for random nouns on the page and you can sift through them but with
seemingly no "weight" given to search results, this can often be a crapshoot. Fo
r instance, the first card in Now on Tap for every one of my e-mails is "Ars Tec
hnica," because it's part of my e-mail address. But of course, what's important
to me is usually the person's name at the end of the e-mail, and any times, date
s, or locations in the body. For our Bitcoin article (this article loaded in Chr
ome), the Bitcoin card was last. The word "Bitcoin" was in the article six times
, and it's the only search term in the title! It should be first.
Sometimes the feature returns nothing at all. In Google Hangouts, when we opened
Now on Tap in response to the question "How is the weather now?" we got... noth
ing. We almost think it would be better to have some kind of overlay pop up wher
e you can highlight the thing on the screen you want to search for.
Now on Tap isn't just about pop-up cards. The "OK Google" voice commands will no
w use information from the current screen, too. This allows you to ask contextua
l questions like, "How far is this from here?" when looking at a place in Yelp o
r Foursquare. Google is (usually) smart enough to figure out that "this" means t
he location currently displayed in the app. When looking at an address in an e-m
ail, we could just say "navigate there" and Google Maps started right up and sta
rted navigating. When it works, it's a great use of context.
Other times, though, Now on Tap doesn't use enough context. We asked "how old is
he" when watching a Stephen Colbert video and were told "73" the age of Stephen H
awking.
One of the downsides to Google Now on Tap is that the opening gesture long pressin
g the home screen used to belong to Google Now. There's no way to quickly open Goo
gle Now from any screen anymore. You have to go to the home screen and swipe lef
t. The gesture also once allowed you to have a third-party home screen and still
easily access Google Now, but you can't do that anymore, either. One option is
to disable Now on Tap completely, and then the old Google Now gesture will work.
Like Google's Search engine, Now On Tap should continually get better, and it's
still early days for the new interface. Algorithms will be tweaked, apps will be
tter support it, and Google is free to update it at any time since it's just par
t of the Google app. At the moment, though, it's hard to "trust" the feature to
do something useful. You're either pleasantly surprised when it works or disappo
inted when it doesn't. Even the most positive of these outcomes, "surprise," isn

't really what we're looking for in a computer interface. "Reliable" and "predic
table" are qualities that are more up our alley, and Now on Tap isn't any of tho
se things yet. We often try it while thinking "I wonder what this will do," but
so far it's been hard to integrate it into any kind of workflow.
The Assist API and Google's continuing commitment to user choice
Enlarge / The assist API settings.
Ron Amadeo
Google Now on Tap is made possible by the "Assist API," a plugin system that sen
ds designated "Assistant" app information about what the user is currently looki
ng at. This data is described in AssistStructure.ViewNode and AssistStructure.Wi
ndowNode. Among other things, it includes all the text in the current view, not
just what is currently on-screen. The assistant app only gets data about the cur
rent view if the user long-presses on the home button it's not a passive scanner.
Still though, if anyone builds an alternative assistant app, we'd imagine users
will want to be very careful about which app they choose as an assistant.
If all of this sounds like a privacy nightmare, the assistant feature can be tur
ned off in the settings. Head to Settings -> Apps -> Configure Apps (the gear bu
tton) -> Default Apps -> Assist and Voice Input and turn off everything. Here, u
sers can also set which app has access to the Assist API (there can only be one)
and pick between sending the app text-only or text and a screenshot.
Sending the assistant app your bank data would be pretty dumb, so developers hav
e a way to opt out of the assist API. Setting the "FLAG_SECURE" flag will stop d
ata from going to the assistant app when the user long presses on the home butto
n. The problem with opting out on a developer-by-developer basis is that your da
ta security is up to the developer. It would be nice to let users take initiativ
e and block the assistant API from certain apps on their own. A user's security
and privacy should never rely on an Android app developer being quick to update
their app. Put the user in control.
Developers can also send extra data to the assistant if they want by using the A
ssistContent class. While Google Now on Tap will take all the on-screen data and
try to figure out the context on its own, the AssistContent class lets develope
rs do a lot of this work. It packages up data to hand to the Assistant app at th
e user's request. We imagine that this would be good for apps that don't follow
the standard Android framework layouts Now on Tap uses to determine context but
still want to play nice with the feature.
With as much flack as Google gets from the FTC and other regulatory bodies for r
estricting rivals' access to Android, the Assist API shows Google's commitment t
o user choice. While developing Google Now on Tap, Google could have used a bunc
h of obfuscated, undocumented APIs that lock the operating system into its propr
ietary solution. Instead, it developed an API alongside Google Now on Tap.
Google added a bunch of hooks to AOSP and then made the most of those hooks, but
any company is free to use them the way Google did. Users can also change from
Google's solution to any other assistant app in the settings. So if Microsoft ch
ooses to make a Bing or Cortana Assistant for Android Marshmallow (or expand its
current Bing Snapshots functionality), the company is free to do so and users a
re free to choose it.
Every part of Android
hird-party app can do
but any app can plug
ext-to-speech engine,

is like this. Google takes over the home screen, but any t
that. Google is the system-wide, always-on voice provider,
into the hotword API. You can replace Google's keyboard, t
SMS app, browser, or phone app.

Granted, Google certainly gets a home field advantage. The platform APIs are dev
eloped alongside Google's proprietary apps, so they are created to do everything
the app team needs. The APIs are also usually developed in secret, giving Googl
e a head start. And of course, Google gets to be the default. Despite all of tha
t, Marshmallow offers more third-party compatibility and user choice than any ot
her mobile OS can boast.
PAGE: 1 2 3 4 5 ... 11 12 NEXT ?
READER COMMENTS 183
Share
615
Tweet
526
Google
1264
Reddit
1523
Ron Amadeo / Ron is the Reviews Editor at Ars Technica, where he specializes in
Android OS and Google products. He is always on the hunt for a new gadget and lo
ves to rip things apart to see how they work.
? OLDER STORY NEWER STORY ?
YOU MAY ALSO LIKE
Google posts Android 6.0 Marshmallow system images for Nexus devices
Paranoid Android redux:

going dark with Silent Circle s Blackphone 2

Marshmallow updates start rolling out to older Nexus devices next week
Liveblog: Google s 2015 Nexus launch event
Huawei s first Android Wear watch is a beautiful yet basic timepiece
Android Pay Google s second take on mobile payments rolls out today
A billion Android phones are vulnerable to new Stagefright bugs
SITE LINKS
About Us
Advertise with us
Contact Us
Reprints
SUBSCRIPTIONS
Subscribe to Ars
MORE READING
RSS Feeds
Newsletters
Visit Ars Technica UK
CONDE NAST SITES
Reddit
Wired
Vanity Fair
Style
Details
VIEW MOBILE SITE
2015 Cond Nast. All rights reserved
Use of this Site constitutes acceptance of our User Agreement (effective 1/2/14)

and Privacy Policy (effective 1/2/14), and Ars Technica Addendum (effective 5/1
7/2012)
Your California Privacy Rights
The material on this site may not be reproduced, distributed, transmitted, cache
d or otherwise used, except with the prior written permission of Cond Nast.
Ad Choices