>> hogue: good evening everyone. glad you guyscan all make it. so let me talk about what i three months ago decided to call the structuredsearch engine when [indistinct] asked me for a title and an abstract for this talk. i actuallykind of debated a bit, you know, what i was actually trying to talk about here. as hementioned, it's--what we try to do in my group here in new york research quality is try toreally understand and, you know, understand the queries, understand the documents, tryto put them together. structure isn't a great word for that. it has a lot to do with whatwe're doing but, you know, it's not perfect. [indistinct] call semantics search engine.this wasn't--this wasn't a perfect year. i
mean, semantics is kind of labeled with, well,semantics. you know, the people in semantic lab, other things like that, it means a lotof different things to people. the understanding search engine, it doesn't really have a goodring to it. intelligent search engine, that's kind of what we're trying to do, but not perfect.that one's taken. so as we often do in this kind of situation, i turn to gary larson toexplain my feelings. i love this cartoon. i mean, basically--you know, i think thisis actually kind of what--what's happening with search engines today, right? we wantto say something. we want to ask a search engine a question. we want to, you know, explainourselves, want to give a lot of background context and things like that and, you know,we wish we could do this. but really, all
the search engines we're really hearing isginger, ginger. you know, it's not totally getting what we're trying to say. a funnystory actually happened just yesterday on our internal quality mailing list. someonebrought up the query. he's trying to figure out what the sixth lane on a running track,what the length of that was like an olympic-size running track. this is--this is the guy thatbasically wrote our index, you know, while we're indexing serving system in mountainview. so he tried the query olympics--olympic track distance. he tried the query track distance--olympictrack running distance. he tried a whole bunch of these queries. he was doing ginger, ginger,ginger, right? he was--he was basically giving, like, these really broad queries and hopingthat google can magically come up with the
answer. it turns out if you put in the query,how long is the sixth lane on an olympic running track? top answer, it's right there. so it'sso bad that even the best engineers at google have been trained to say ginger, ginger, gingerto the--to the search engine. but it reality, like, we're actually getting to the pointwhere google can try to answer some of these questions about a more deeper understandingwith longer queries, with a more deeper understanding on the documents and the content that go withthem. so today i want to talk about some of these efforts that are going on. most of theseare going on in new york on the search quality team here. key topics i guess: what can weunderstand about the world? it's kind of like where we started. like, what things are inthe world? can we better understand the queries
that we're getting from our users? can weunderstand content of the web, documents, user reviews, things like that? and then probably,we can put all these together and actually let google do some of the work for you andyou kind of put all this together so that you wouldn't have to work quite as hard toget the answer to your questions. so i won't read it but, you know, we're kind of go througha couple of technology result in here and how we try to answer these questions. so thefirst question is, you know, can we actually understand what's going on outside? the searchengine shouldn't operate into a vacuum. you know, we should be actually understandingthat there are real things in the world. it's not just documents and n-grams and so on.so as many of you quite have heard we acquired
a company called freebase over the summer.they build a structured database of everything in the world. you know, this is a graph. it,you know, allows links from between things so we know that, you know, bono and the edgeare members of u2 and so on. so for those that haven't seen freebase before, it's adatabase of entities as i've said. it's got connections. it's got properties. it's gotstrong semantics so it's actually a schema for freebase. it knows what a company is.it knows that it has employees and it has a ceo. it's got about 20 million topics rightnow. it started of--when we acquired it, actually about 13 million a week, we made a strongcommitment to making it bigger. we actually are a whole bunch of music data that's beenpublished now. so google's really committed
to making sure that this becomes a canonicalreference point for high quality, 99% precision entity topic data on the web. this data isall--this data is all publicly available. this is an advertisement; its creative commonslicense. you know, you can--you can do whatever you want with it attribution to freebase.has great apis; i mean, it's got a very simple new java script query language. and there'ssome very good tools that they have, this thing called gridworks, which is recentlyrenamed google refine, where basically pulling in new databases of information and reconcilingit into freebase. so if you've got a great database of ancient mayan art or somethinglike, freebase doesn't know about any of it, we'll give you the tools to actually importthat and make sure that freebase already knows
about some of it. you're reconciling it properlyand merging it properly and then you can make this better for everyone. so just sort ofgive you a little bit of an example of the things that freebase knows about. you know,it knows about buildings. it knows about molecules. it knows about aztec gods. you know, thisis--this is a place where we start to get further away from standard kind of databasesof everything like wikipedia. it knows about candy, international candy, u.s. candy. itknows about art, of course. just a quick example of how freebase represents information. queryknows that there's an entity called blade runner. i know that it's a film. there's aboutother names for it. you can see that it's not only blade runner, but there's a namein russian and various other languages for
it as well. it knows that harrison ford actedin it. you now, the relationship between a movie and an actor is not a simple one. they--it'snot just an actor. it's the character that he's playing. freebase represents this butsome call it a compound value type, which is a way of kind of mediating between an entityand kind of complex information. so it's not just a simple [indistinct] store. it's a--it'sa way of actually representing complex real information about the world. so freebase iscool. we can do a lot of stuff with that. you know, we can build interesting searchengines but--like i said, our goal here is to actually have a database of everything.we want to know about everything in the world. so, you know, 20 million entities, that'scool. you know what's really cool? a billion
entities. so how do we get to a billion entities,right? i think the really--the only way that we can think of, and we can do this manually,you know, we can kind of walk through and try to have each person to contribute, youknow, their own 20 entity or the thing that they know best. but really, we're going tohave to do something automatically. and there's a lot about what we concentrate on here atnew york. so what can we actually extract out of documents? the other simple stuff,things we can recognize with patterns, dates and times and measurements, phone numbersand things like that. some things are harder because they're kind of ambiguous; thingslike locations. you know, starbucks has many locations, which one is this document we'reactually talking about. you know, people obviously
very ambiguous; can we actually recognizewhich people are being discussed in certain documents? and then even more complex typeslike [indistinct] with freebase, you know, factual data like what are facts about anentity? so here's an example of a page that is an entity, this floor standing speakerthat freebase has no idea about, right? this is a long-tailed entity. and this is actuallya pretty good page. i think we can pull a lot of information out of this. we [indistinct]the manufacture and the part numbers and so on. you know, it's a good description of thisthing. i would love to create an entity out of this thing. if you scroll down the page,there's a really nice table here. you know, it's got all kinds of [indistinct] information.it'd be great to get this stuff in. so our
team here in new york works on pulling thisinformation out. we use a whole slew of techniques. we look at kind of--we look at tabulated data;a data that looks like it's organized in some way. we look at a data that looks like attributesthat we've already heard of. so maybe we've already heard that weight is an interestingattribute. lots of things have weight; freebase knows that. so whenever we see in weight followedby a measurement, let's pick it up and put it into the--into the database. even crazythings. once we pick up a couple of attributes, we can get a pattern. we can introduce a wrapperfor the data and actually pull out things like input impedance. you know, that's somethingthat freebase might not know about. we might not have it on our schema. that's still usefulinformation. so the approach here--just not
going to go into too much depth. that probably--thistalk is going to be a huge fire hose and then you're going to probably have to join googlein order to find out the real details but... so the approach here as always the googlewe have a huge--we have a large amount of scale. we know the whole web. we're goingto go for a very high coverage system here. we're going to give up something in precision,but we're going to hope to get it back by aggregating data from lots of sources. [indistinct]tables. we'll look at, you know, things like attribute common value. we know certain valuestakes certain types. so depth is always a measurement of length. and if we can recognizethe measurement of length and the word depth is nearby, it's possible--it's quite probablethat someone is trying to describe an attribute
here of some entity. we look at things like--youknow, standard techniques like page segmentation. we look at wrapper induction to kind of inducethe pattern for what we're seeing on the page. so all of this obviously, as i said, not muchdepth here but lots of machine learning going on behind the scenes to produce this table;you know, lots of--lots of evaluation. we want to find out how we're doing, kind ofiterate on information we're seeing, what's working, what's not. and we try to eventuallybuild up this database. so you notice that the bottom here by the way is just to pointout the high coverage, low precision mean here. you know, it's not just the good stuff.we've ranked this and we're doing pretty well. i think i'll take questions at the end. isthere anything very particular right now?
>> yes. the number on the left of the--onthe [indistinct], the left hand [indistinct]. >> hogue: yes.>> [indistinct]. >> hogue: sorry. so the--the question wasabout the number on the left inside. so that's just the confidence that we assigned. it's--youknow, you can think of it as maybe like a probability of being true although that mightbe glorifying in a little bit. so as far as what i'm saying where--you know, this is justthe cream of the crop here. if you look down a little further, you know, it's startingto get a little sketchy as we go to lower and lower confidence values. but, you know,these things all kind of look like attributes and values to our system. but thankfully,we've been able to kind of pull out from the
[indistinct] here at the top and that becomesthe useful data that we have. so that was--that was the first fire hose. that was kind ofsome how we understand the world outside the web and what can we do to kind of build upthis database of everything and hopefully make it to a billion facts. we're not--we'renot quite to a billion yet but we're trying. so the next--the next step is can we actuallyunderstand queries? you know, can we understand what users are saying when they're tryingto look for document information? and so--and this man want to talk a little about questionanswering. so this is actually my started project at google, you know, six years ago.still doing it. it's a--it's a tough problem. as the [indistinct] it's a great shot. snappedby one of the guys in the team on the subway.
you know, google--well sometimes these aren'tfound on google, but we're hoping the ones that are found at google that we can surfacethem in a way that people can find useful. so just to give you an example of what i'mtalking about, i'm talking about this one box here at the top. if you ask, when wasmartin luther king jr. born? we want to surface the answer right away and kind of give youthe right correct answer. the team is also responsible for highlighting the answer inthe web search snippets and especially if we have lower confidence that this thing isright. we want to still give you some indication that maybe this is a date that's useful foryou. so i'll take you to a quick work example of how we understand a query like this. sothe first ting we do, as i mentioned before,
we have a lot of systems here for understandingsimple text. we call them annotators; things like dates and times and measurements andso on. one of them is very good at recognizing names. so first clue we have here for thequery is that it mentions a name. that's a good piece of information. once we've gotthat bit, or even if we don't, we next try to split the query into a thing that the useris asking me about; we call it the entity and an attribute. you know, what propertyof the entity are they looking for? if we can't find that, we're done. you know, thisis probably not a question. but if we do, we're going to try to figure out how to--howto retrieve a good value for that. the next thing we look at, you know, we have a largedatabase of entities. we know something about
how to relate it. do we know any other namesfor the entity being mentioned? in this case we know that martin luther king is--jr. isalso referred to as martin luther king often mlk and so on. one of the things--you know,when somebody ask for someone when someone is born, what are the ways could our databasebe representing that same value, date of birth, date born and so on? we look at the wholequery to try to give us some other clues as to what the answer might be. we have a bunchof, you know, to be honest regular expressions that kind of say "well". and if you say "when","was", it's probably looking for a date. even if we didn't have that, because we have thislarge database and we can kind of do a lot aggregate analysis, we know that date of birthare born often looks--often has the value
of a date. so we're--we think it's probablya date. "where" can also mean where they were born. [indistinct] a little bit of confusionabout that. but we're going to try--we're going to favor dates in this example. so nowwe go [indistinct] database. as i mentioned, we've got a big table full of informationlike this. you can see that we've got a couple of potential answers here. the birthplaceis there, also the birth date. and we're going to go in now... sorry. the big insight hereis that we're not just going to look at the table. if just look at the table, lots andlots of query would match and we'd be able to answer pretty much any query with somethingout of the table. as i've shown you before, there's a lot of low confidence values inthere. but the big insight here is we actually
look at the top search results. google.comsearch is very good at delivering on-topic information for pretty much any query. sowhen you say, when was martin luther king jr. born? chances are the top--if it's a question,the top documents have the answer. and so in this case, we go--we look up our suggestedanswers in the documents and try to figure out if they appear. in this case obviously,the dates appear. you'll notice also that the birthplace appears as well. but becauseof two factors, first, we find the date more often, and second, we are expecting to finda date; when is it happening is, you know, we actually answer the question correctly.we got a lot of collaboration at the top search results and we try to get the users some waysof actually backing this up especially--we're
probably going to be wrong five, 10% of thetime. we want to make sure these are kind of [indistinct] checked, also figure out wherethis answer came from. so that was the end. i know there are quick fire hose, anotherproject we're doing here in new york about question answering. so the next topic i wantto talk about a little bit is what we can do to understand content. so we've lookedat the world outside. you know, we can understand entities now. we can look at queries. we canunderstand how users are asking questions and we can look at--now we want to look athow we actually understand the documents, the content on the web. i'm going to predicta particular slice of this and we've already talked a lot about extracting informationfrom documents and how we understand that.
but i want to talk a little about sentimentanalysis. so sentiment analysis is the field where we're looking at. can--how can we understandwhat users are saying about let's say a business, a product, a person. is it positive? is itnegative? are they happy about their experience? are they upset? you can put it on anotherway. it's a different [indistinct]. you know, erick and his googly eyes or double face palm,right? i mean, like, what are users really--how are they feeling about this? so just an example.this is one of the things that's actually out there on google.com today on the placepages which is part of local search. we have a listing here for the carnegie deli. andyou'll notice that we're actually summarizing the reviews for the carnegie deli. there's1,500 reviews almost for carnegie deli. and
if you really want to get a good feeling forwhat's going on, should i go there? is it really, well, it's cracked up to be, you canread all 1,500 reviews. but if we can we can, we can actually summarize and try to giveyou a feel for it. one thing you'll notice is that the star rating for the carnegie deliis three and a half stars. that's kind of like an aggregate average of all the reviewtext that we've picked up. and the snippets that we put up here, we call them frankensnippets because they're kind of, you know, sewn together. they're trying to be balancedin the same proportion as that three and a half-star rating. so there's good stuff. obviously,great food, great service. some not so good stuff. some people think the waiters and staffare unprofessional, and you pay the price
for it, too. so we're going to try to balancethe reviews. somebody has five stars. i'm going to try to give you mostly positive reviews.one star, mostly negative reviews. now i'm just trying to give you a good feel for it.you want to get into more detail, we also try to parse out what aspects, what thingsare people discussing about this business. you know, in the case of a restaurant, thesubstandard ones. obviously everyone talks about the food and the service. but the carnegiedeli is known for its corned beef. so we actually are able to identify that, pull out the reviewsautomatically and determine that corn beef is a pretty big hot topic. everyone's talkingabout it in positive or negative terms. so obviously dig in even further and find outall about the desserts and so on. and you
go even further. i mean, we're really tryingto give you a tool to really dig down, why is it that everyone's talking about the cornedbeef? what's wrong with the service? why is that, you know, almost half red and so on?so that's just a little bit of the set up. this is why we want to analyze sentimentsand summarize. so sentiment analysis is a great field for machine learning and nlp.there's so much text. it's such, you know, an interesting, you know, kind of deep naturallanguage problem. we have to deal with a whole bunch of different issues here. obviously,this is a basic one which is, you know, what is--what is a positive sentiment? what isnegative? what words represent positive stuff? what words represent negative? these aspectsas i discussed, what are people talking about?
you know, these deeper natural language problems.what happens when somebody negates a word? you know, "it wasn't the best." you know,we don't want to give it credit for positive just because the word "best" appears there.and as well, we don't--we also don't want to apply the negative sentiment to the latterhalf of the sentence. and then finally; scope. you know, often times people say things--excuseme--like "we came here because we couldn't stand the lines of the other restaurant."we don't want to give negative credit to this restaurant because of its lines. it's not--it'snot the right scope. they're talking about some other restaurants. so these are the kindsof things we're dealing with if we want to accurately summarize information from reviews.so i may not go into all of this. i might
just go through a couple of worked exampleshere of how we deal with the positive and negative words and how we deal with negation.so this positive and negative problem we call classification--sentence classification. sogiving a sentence, giving a review, is it positive? is it negative? you know, how isthe user expressing their opinion? this is a pretty broad goal. this is google, right?so we want to be--we possibly use this in many domains. we don't want to be just doinglocal. we want to do products, news, people, whatever we can do. it's got to be international.we have a ton of traffic coming in from outside the u.s. and u.k. and the english languagecountries. we don't want to just make a solution that works for english and then stop. it'sgot to be robust. a lot of consonant in the
web is misspelled in case nobody even noticed.you know, it's got--and people don't use proper grammars. and this can't be, like, a reallyhardcore, you know, parsing and so on. it's going to mess up a lot just because somebodymisspelled the word. and then obviously it's got a scale and they were dealing with millionsand millions of documents easily. i want this to be updated everyday. i want to be ableto take reviews very quickly and so on. so the approach just for a classification isto build a lexicon, we call it, which is a set of words that have some meaning associatedwith them; of words that have positive and negative associations. and we want to do thatquickly from a small set of seed. that's how we're going to crack the internalization problem.we're going to say we just start with a small
number of words and we'll be able to expandupon them. so, really quick. to build a lexicon you start off, as i said, with a small setof seed words. you know, these are simple things like good and bad, fantastic and soon. about 100 of them actually is enough as it turns out to just kind of give the systema sense of where to put the dividing line between good and bad. and you also take alarge graph of n-grams. so you think of it just a little bit like wordnet. you know,it's a set of words or phrases that have associations between them. in our case, we actually havesomething a little bit more powerful than wordnet. we actually take the whole web corpusthat we compute what's known as a distributional similarity metric between all the n-gramsthat we can find. so it basically says, "what
kind of context do these word appear and howsimilar are those context?" as compute this--the whole web corpus and we end up with a very,very large lexicon, a several hundred million phrases and edges with weights between them.so it's a big graph that kind of says which words are related to each other as far ashow they're used in the language. next thing again about this, you know, there's nothingenglish specific about this. all we need is a bunch of n-grams and a bunch of documentsin that language so we can kind of go ahead and build this graph. then we run to what'sknown as label propagation over this. we start of by labeling the notes that have positiveor negative sentiment and we kind of iteratively propagate those weights through the graphuntil we reach some sort of, you know, steady
state that says this portion of the graphis positive, this portion of the graph is negative, and this portion of the graph, well,we don't know. it's probably somewhere in the middle. and remember, this isn't justwords. this is all kinds of phrases. you know, this is things like truly memorable, right?not just truly, not just memorable, but truly memorable is the fairly positive word, oneof a kind. it turns out [indistinct] is slightly negative. who knew? so a pain in the ass.you know, squeaking, internal bleeding clearly negative. when somebody mentions internalbleeding in the review of a restaurant, you probably don't want to go there. right? sosince this is just a really small sample, the weight associated with this kind of rangefrom negative five to five, you know, it's
just based on how positive or negative wethink that these words are based on the context we find them. so just to give you an exampleof why it's good to use a lexicon like this, it's built from lots and lots of phrases insteadof just a simple dictionary. we can actually tell the difference between nouns and otherkinds of typically non-sentimental words based on the context that they occur in. so thingslike dog. dog is not a particularly sentimental word unless you have one. but dogs barkingis negative, dog friendly is positive, right? self-sufficiency is good in terms of self,but self-servicing is a bad kind of phrase. painstakingly is different than painful. attentiongrabbing turned out to be a good thing. money grabbing turns out to be a bad thing. youknow, and even great--even positive words
like great sometimes mean negative thingslike great expense is not necessarily a good thing. so this is kind of the power of havinga lexicon that's built from lots and lots of documents, lots and lots of [indistinct],you know, different, you know, ways of representing the same information. because the other sentimentkind of task that i want to talk about briefly is negation. how do we actually handle whena user negates a particular word. so they say not great. you know, how do we actuallytell that great is not actually a positive in this case but it's actually negative. sowe could do this for the lexicon, right? it's possible that we'll see the phrase "not great"or "wasn't great", enough in our web corpus that we'll be able to identify as a--as anegative thing. but it's better--it turns
out to be a lot better if we actually builda specific tool for identifying negations. so just to work a quick example. this is areview--a piece of review text here. it's pretty standard. "service wasn't the bestbut the food more than made up for it." and this is what our negation system tries todo with it. so the beginning of the sentence, zero here or green is basically is not negative.so it means that there's no probability that this is a negated part of the sentence. andas we go towards one, it means something is probably negated. so it wasn't the best. sobest is not actually a positive sentiment here. with a high probability, the user isnot talking about best. it's talking about the opposite of best. and then as you cansee as you go down the sentence, the probability
of that is if it's still a negative, it kindof dwindles off. you know, this isn't--this isn't a perfect example here, right? we actuallywant to treat food as a positive thing and we're just barely, you know, kind of makingit down towards where it's probably not negated. but if you set the right thresholds here,we actually do fairly well in identifying negations. so the way we did this, we [indistinct].not me. we took a golden data that we basically hand-labeled about 2,000 reviews. so we hand-labeledthe negations in them. you know, we try to get good agreement between the people thatwe're labeling these so that everyone kind of said the same thing was negated. and webuilt a--what's known--a user technique called a conditional random field, which basicallyoutput probabilities. it takes a bunch of
features and outputs probabilities of whetheror not what you would like to happen is happening in this particular section of the text. andwe trained it based on these positive examples. so just to kind of--to give some of the resultshere, this was published last year on the workshop. and this was actually on the publicbenchmark [indistinct] called bioscope; a bunch of biology papers. this is actuallythe best results that had been published so far. it also has a dramatic effect on ourinternal metrics where we--when we looked at local reviews. i don't know if it's easyenough to see on the presentation, but there's a red line, which is the old system, a blueline, which is the new system. better is up into right. so we did a lot better here. justreally a quick point i wanted to make is,
you know, kind of [indistinct] which is thatgoogle does want it, you know, kind of worked and published some of the--some of the stuffthat we're doing. i mean, it's not just kind of black whole here. we're actually tryingto get out, publish papers at workshops, at conferences and so on. i think the sentimentanalysis is a great example of this. both the lexicon work and the instigation workwere republished this year. and we're applicable. i mean, we're really trying to get the data--theresults out there and share them with the community. just a few more examples of negation;some that worked out well and some that didn't. the underlying bits here are the things thatare being negated. so your negations are hardly a simple problem when detecting sentiment.you know, a simple problem is something that's
not a positive vehicle here. you can see,you know, we do well on certain things like "don't cry for me argentina." you know, that'sa--that's a good example. it's just that it does have some issues. it doesn't do wellon the kind of yoda-speak. it looks--it's very dependent on finding evidence to theleft. it doesn't do, you know, very well to the right. so if you flip--you invert thenegation, we just don't see a lot of text like that. i'm sure we could try a few examplesof yoda-speak but they just don't come up very often in actual reviews of restaurants.so unless people start writing reviews in yoda-speak i think we're all right. okay.so that was the section on trying to understand content, both the stuff that we talked aboutin the beginning about understanding and extracting
information from content as well as understandingcontent at a deeper level of natural language processing with sentiment analysis. so thislast section i want to talk a little bit about, can we actually have google do some of thework for you? we got all this great stuff going on at the back end, but most of it isstill ends up doing a single query and doing one kind of bit of information processingfor you. can we actually do something that actually does lots of query on your behalfand saves you a lot of effort? so it's--maybe this doesn't happen everyday, but maybe there'ssome problem that we take a lot of effort to put together the information that you need.and the question is: can we actually save you some that time on these very complex tasks?so let me talk a bit about google squared,
which was our last product that we launchedabout a year and a half ago. for those who haven't seen a google squared is trying totackle the problem of kind of hard decision making problems that don't occur everydaybut, you know, a kind of high value, right? so we multiply frequency by value. these arestill about as--you know, have the same in factor as a typical everyday question [indistinct]query because they're much more valuable even though they don't occur as often. so thesethings are like buying a car, planning a vacation, choosing a college, things like that. thisis a personal example; my wife and i a couple of years ago were buying a car, right? andwe--this is before squared launched, before it was kind of being developed. so we didwhat probably a lot of people would do. we
actually--you know, maybe we kind of wanteda big car. we've got kids. we actually made a spreadsheet, you know. we put in that spreadsheet;we put a list of cars. you know, we put--and then we were interested in the--you know,the toyota highlander. you know, we were interested in the saturn, you know, whatever. put thatdown on the left side. and there's things that we cared about. we cared about the crashtest rating. we cared about the prize. we cared about number of seats, things like that.but those [indistinct] the top. and then we did a search for each combination of thosevalues, you know. we actually went through and we found out what the crash test ratingwere in each of these cars. and the problem with this is that there's no one [indistinct]that quite does this exactly where you'd want,
that it's flexible enough to have both crashtest rating and fuel economy and, you know, like the prize let's say. you know, like--youknow, some [indistinct] has a lot of information. it's not as configurable as we wanted. andso we ended up using a spreadsheet and kind of manipulating the data ourselves. so theseare the kind of problems that google squared is going after. the magpie here, we choosethis as our internal codename magpie because it's a bird that collects lots of little bitsof, you know, foil and hay and garbage to build its nest. and, you know, some good,some bad but, you know, it's a process of collection and kind of aggregation which isvery similar to the kinds of tests that we were going for. so for those who haven't seenit, google squared is kind of an example.
you type in one query [indistinct] and googlesquared goes out and builds this whole table for you. it's got a list of things. in thiscase, baseball players, pictures, descriptions, facts about them. [indistinct] just peoplethat can do antibiotics. you can do things like cheeses. actually we're going to tryand do a quick demo and see if this works. so to see it in action--so this is just anexample. we type cheeses, it builds this entire square on its own. one thing to know aboutgoogle squared is that it's not trying to do all the work for you. we know in the caseof information extraction, [indistinct] domain information extraction about the worlds here.as i said before, this is a high coverage, possibly a low precision kind of, you know,domain. so we're actually going to only show
you the things we're pretty confident of butwe don't--we actually aren't sure that we have the right value. we're going to eitherhide it or, you know, kind of dim it out a little bit. and we're going to work with theuser to try to help them understand what information might be right for their task. so, yeah, iliken this to google.com search. if google.com search were perfect, every query would beunfeeling lucky, right? it wouldn't have 10 results. it's not, right? i mean, like you--it'swrong. it's wrong a lot as it turns out. trust me. so the idea is you get lots of links.you get a lot of feedback. and even for things like, you know, what effectively questionanswering here? we want to give the user some flexibility to kinda correct things. so, youknow, maybe--you know, i want to actually
go and look at wikipedia and find out this,you know. i believe wikipedia that [indistinct] and that's fine. so i want to include thisin my--in my squared. and google squared learns from this feedback. we kind of, you know,look at users that are adding and deleting rows, users that are correcting values, changingvalues, and these are all editable so i can go in and just change. i don't--i mean, idon't like this one. you know, squared is good at kind of adding new items. i noticedthat swiss wasn't in here. so maybe you want to add swiss cheese. and it'll go off andit'll try to pull in new values for each one of this and kind of fill in the table. swisscheese as it turns out [indistinct]. you can also--you can also add columns here. so let'ssay that i want to know what kind of wine
to pair this with. so this is a tough--thisis a tough query. google squared doesn't really know exactly, you know, what i'm talking about.this is probably is in a standard attribute in freebase. you know, we can't just go toa high quality source of information. but we do have some--you know, some informationhere. you know, we picked out the wrong part of the value but, you know, champagne. thatseems like a good value so i'll put it in here with my [indistinct] computer [indistinct].all right. you know, this one--[indistinct], beers and [indistinct]. so this is the kindof process. we want to give people away a very quickly pulling in information and buildingthis--you know. we all we want to decide what cheese [indistinct] in and kind of figurethis stuff out. google squared, where do we
start from? what are the key premises here?you know, why are building this? so--and then how are we going to build it rather? so onething we noticed as i--as i spoke about before with the question answering and so on. websearch is a great--a great tool for information extraction and trying to find the right information.we do a research at google.com, it tends to get relevant documents. enough that we canuse it, right? so we're not going to just use it for a database. we're actually goingto use all that information and store it inside google search and all that pent-up experienceand so on. one--another point here is that a key step to making a decision, you got tocollect data, right? i mean, if you're buying a car [indistinct] college, you need lotsand lots of data and you want to kind of put
all that together. another premise, as i mentioned,is never going to be 100% accurate. we're not going to even claim this 100% accurate.really lucky if we get 70% accuracy on open domain information extraction. that wouldbe world class. so we're going to make a trade-off here. we're going to try to go for a highcoverage, but we're going to pry users with tools to correct their data when they--whenit goes wrong. and finally, can we actually make use of this search engine here and canwe do a lot of work on behalf of this, or can we save them all by typing or puttingeach individual query to try to get the squared--our table built? and this is kind of fun fact.i mean, google squared--to build that square [indistinct] cheese, it [indistinct] 200 queriesto google.com and [indistinct] all that information
together. so just to talk a little bit abouthow this gets done: first phase, you got a broad query, let's get a list of names. thisis the thing that goes on the left side; what kind of cheeses are there? you also want toexpand that list. so if you've already have some cheeses, what other cheeses might beinteresting? what kind of attributes are interesting for this? i'm going to go into this in a littlemore detail on the--and finally, can we actually pull out a value to put into the square. sofirst step, as i said, finding a list of names. so the approach here, we want to take [indistinct]cheeses and go to something like brie or gouda. we're going to do a search. as i said, thatseems to be a pretty good tool. here, look at the search for cheeses. these are prettygood results as it turns out. now this result
has a really nice comprehensive list of cheeses.so as this one. one thing you also notice down here is that these are really comprehensivelist. this is not actually giving as much information about which cheeses are actuallyinteresting, which are the popular cheeses. you know, the wikipedia entry has on the orderof 1,000 cheeses listed or something that. and i'm just trying to figure out what i wantto search [indistinct] with my wine, this is probably not the best way to go; you know,go about it is just going to clicking on each one of these links. so we're going to tryto help these organize these things because that requires some ranking. so we got a candidatelist of maybe 1,000 cheeses or things that might not even be a cheese and we're goingto try to rank these things. so one approach
is to try to get more list as it turns out.not less. so we run other queries. we run list of cheeses, kinds of cheeses, top 10cheeses, popular cheeses. some of these are very good for getting comprehensive list likelist of cheeses. some of these are good for getting popularity of cheeses. so, top 10cheeses is actually a great way to get a list of people's favorite cheeses which gives ussome ways of separating out the interesting stuff from the kind of long tail. there'salso a lot of user feedback that we can use here. so squared users, as i've said, edita lot of tables. so if someone has done this query before, you know, possibly they've added--that'sa pretty good signal to us that, you know, this is probably a good cheese or whateverwe're talking about. web users also type in
the query of cheeses a lot as it turns out.we can look at what they type next. do they refine cheeses to brie or gouda? it's muchmore likely they're going to do that than they're going to do something much more rare.and then we just have the raw popularity. more pages contain brie, especially pagesthat contain less than danbo, a type of danish cheese, that i just kind of randomly pickedfrom the list that i've never heard of before. so that's kind of how we--i get the initiallists of names. we're trying to do this two-step process of pulling out list and then rankingthem. now we want more names. we want to be able to suggest additional ones especiallyif you've done some modification of the squared and you've added your own things, maybe you'vegone down a particular path, you're just using--you
know, putting in european white cheeses orsomething like that. can we actually suggest more things that fit in to this category?so in this case, we use something very similar to google sets; same basic algorithm. we goout and we look a list on the web, [indistinct] offline and we say given that you have severalitems, what is the most likely other set of items that would be in the list, anyone onthe web? so given that you've already seen brie and gouda, you look other list on theweb that say brie and gouda, what other--what other items tend to occur in the same list?and it basically gives us a probability and we use that to kind of rank suggestions forthe pop out at the bottom that kind of suggests other things, you might want to add to yoursquare. next, we want actually have some things
to go along the top. so i mentioned descriptionis kind of given. every square is going to have an image and a description. but every--otherthan that, there's almost no commonality between domains as far as what kinds of attributesit might be interesting. if people want [indistinct] birthday for cheeses, you want to know their--youknow, where they come from and so on. so obviously the first thing you think off, go and lookat the fact table. you know, you got this big table. hopefully, a billion of facts.what [indistinct] does it say about brie? this [indistinct] is actually pretty good.you know, it's got things like source milk and aging time which turn out to be prettygood. it doesn't always work. cheddar turns out to be a fairly conflated name. you know,we don't have, you know, kind of a strong
idea of which one of these that might be.it turns out it's a town in south carolina, another town in united kingdom; also apparentlyan album--no, a band i guess or maybe an artist. do we know who chadder is? so this doesn'talways work. i mean, we can aggregate. that helps. i mean, you look a lot of cheeses,see what their attributes are. but i think we want some sort of signal here to kind ofhelp out. so there's a theme here. second approach is go and look at web search. sowe actually take each one of the cheeses and we do a search on google.com. we do a searchfor brie. we do a search for gouda. we do a search for--and we look at the originalsearch for cheeses as well. we look at the tables in that search. and it turns out thatthere are certain attributes that kind of
mentioned over and over and over again; thingslike texture and country. and this is the way of kind of narrowing down the context,right? not only that web search kind of--tend to be on-topic most of the time for this queries,but it also kind of allows us to aggregate across many, many different queries of thesame type and disambiguates. so even if there's some ambiguous things in your list like maybeyou're looking at cars or car manufacturers and you've got ford, then you might mix itup with ford the president and get date of birth, there's likelihood of that happeningover and over and over again for every individual car manufacturer is fairly low. and so bydoing this throughout the entire list, we end up with a good set of attributes. in thefinal step, i basically already talked about.
when you want to find the value to go intoto cell, that's question answering. and actually what ended happening was we took google--launchgoogle squared about a year and a half ago, that launch from the labs, that was great.we got a lot of good feedback for it, but it's labs. you know, we only got a few--youknow, tens thousands of users per day. we want to get this stuff out at google.com sowe took the same back end that runs the cells for google squared and we put on google.comand that's now our question and answering system. so this is, you know, an example ofwhere labs is kind of the breeding ground for the good stuff that comes out next onweb search and we got a lot more good stuff coming down the pipe from here on out. sojust want to talk a little bit about what
we've learned in general from all of thesetasks. so first, and i've kind of reiterated this several times now, web search is reallypowerful, it's a great way to do information extraction. google.com knows, you know, ifyou say cheddar, it knows that you're probably talking about the cheese even if broad, youknow, strings--you know, the strings in our fact database sometimes are talking abouta town or a musical artist. web search kind of stays on topic, it helps us stay on topicfor the different things that we're looking for. it also is very, very deep. it, you know,it has things for the long tail, has lots and lots of documents are all on topic forgiven--for a given subject. another thing is scale. as you saw from question answering,from extraction, as we saw it with the way
that we build [indistinct] analysis, scalelets you aggregate. you know, having this many documents, having this may queries, havingthis many machines allows us to do things that kind aren't able to be done anywhereelse. we're tackling kinds of these kind of nlp and information extraction problems. idon't think it could really be done in any other setting. another thing is that i talkeda lot about, you know, coverage and so on and the trade off, but it turns out that precisionis actually really key. we ran a survey at the top of google squared to kind of ask usershow they were feeling about their experience with squared and whether it was useful andwhether they saw their task. we did a release at one point shortly after we launched thatimproved our measured precision internally
on our evals. about 10% were absolute. itturns out that the satisfaction of the users also went up by 10%. so, it's, you know--it'snot--you're not doing this in a vacuum. it's actually used out there and improving somequality, actually have substantial improvements in user satisfaction as well. another thingis that coverage is very, very hard. you want to get into the tail, you want to understandthe whole web, you want to understand the whole world but, you know, it's a very, veryhard, difficult thing to go into the tailing but it's critical. these are some of the queriesthat we've gotten on google squared. people actually typed in titanium rings, design software,artificial tears, you know, these are queries that people really want to get and build asquare out of and, you know, we need to be
able to find this for them and find out waysand help people solve their problems. another thing we learned and--probably from the [indistinct]is that you fail, you can ask, like, ask the user. don't--you shouldn't be shy about, youknow, kind of being perfect and right every time. that kind of makes that hard, brittleexperience that isn't likely to succeed, you know, every single time somebody tries somethingslightly outside of the domain of what you plan to build it for. so you should buildsystems that are kind of robust to user feedback and accepting of user feedback so that theycan correct and you should learn from it. and the final thing, another pitch, this is--thiskind of work on open domain information extraction. it's hard, you know, but it's pretty excitingand i think this is the place where we can
make a lot of impact by having the scale thatgoogle has. so i'll repeat artie's pitch, we are obviously hiring. and i wanted to takequestions. if you could use the mics actually, that would be great; that way the video andthe--everyone else can kind of hear then. >> freebase is an on the acquired technology,right? >> hogue: that's correct. yes, from freebase.there was a company called metaweb. >> that was danny hillis' people, isn't it?>> hogue: danny hillis founded it, yes, yes. >> yes, thanks.>> hogue: no problem. i'm going to have to bribe you with a flag?>> are you going to take the data that users enter into squared and put it back in to searchengine?
>> hogue: we do use the data for feedback,yes. we look at user corrections. we look, especially at--when adding moving rows, whenthey're adding columns, when they're correcting values. obviously, it's a source of spam,right? i mean, people can, you know, go on and put that, you know, their name is thecurrent president of the united states. that's--you know, it's a great wish, but it's not true.so, we have to look at it in aggregate and we always have to compare to signals that,that exist outside of squared but it does turn out to be a very good signal, you know,and you can actually identify spammers. they're the ones that fill the entire column in withtheir name. you know, they're not being subtle as it turns out, so. yes?>> hi, do you guys do any normalization of
the data that goes into the squares, likestructural [indistinct] or anything along those lines or do you just pull it straightfrom the web? >> hogue: we definitely do normalization;nothing kind of hand created like at the source necessarily. but, for instance, i mentionedthe [indistinct] tiers that we have; we understand measurements, we understand dates, we understandlocations even, so if we see somebody refer to london and we see somebody refer to london,england, you know, we know that there's a high probability that those could be the samething in the right context. or if somebody says, 39 inches and somebody else says a meter,we can kind of normalize those things and we do that kind of normalization especiallywhen we have some information about the semantics
of the string there. that's being used.>> okay. >> hogue: yes.>> okay. >> hogue: hi.>> i have three questions, one of which is a follow up on the normal--on the normalizationquestion. >> hogue: sure.>> do you guys do some totalization in your extraction algorithms? that's question one.>> hogue: so what do you mean by totalization? i'm not quite...>> so like root of a word. >> hogue: sure. so we do some of that especiallywith getting lists, you now, and forming the queries. so obviously, we--if somebody sayscheeses, you know, we need to know it's the
same as cheese and so on, so we do get rootsin that sense. i'd say that that's probably the main extent of that kind of parsing. wedo a little bit of sentiment analysis as well with the aspects, so if you're talking aboutservice and servers, we can kind of collapse those two aspects in the same kind discussion.but yes, that's the probably the extent of it.>> okay. next question, how do you--what's your experience with bigrams and trigramsand how did you weigh the two? so would you look at the bigrams more than the trigrams?or you would see if you can find enough data for three words, sequentially, would you usethat more? >> hogue: yes, it's a good question. so, i'mnot so sure if everybody--can everybody hear
the questions up through the mics? [indistinct].so there is a bit of light tuning there. obviously, there's a--but it comes out more in actuallyoverall frequency in the corpus, right? so we do some tfi/df. for instance, like thephrase "a great knight" turns up less often than 'great' or 'a,' you know, because it'sjust a more complex term and obviously, it's going to show up less often. so, we do thatkind of normalization. we have found out that it does help sometimes but, you know, actuallyyou should probably talk to isaac sitting on the steps out there [indistinct]. he'sthe tl for the sentiment team. if anybody is really interested in that kind of nlp,he's a great guy to corner after the talk. >> cool. and last question is, are there anypapers in the field that you found useful
in classification ir?>> hogue: i--i'd actually refer you to isaac again. so i think, you know, probably checkthe references from the papers listed in the talk.>> all right, cool. thank you. >> hogue: yes, no problem. let's go back towork. >> hi. just wondering how you measure usersatisfaction in search? >> hogue: that's a good question. so we actuallyconstruct a survey that basically asks both broad questions about the overall experiencewith something like squared, so, you know, "did you solve the problem that you were tryingto tackle?" as well as very specific ones like "did you use feature x? did you add arow to the square? did you know that you can
add a row to the square?" we also ask questionsabout specific updates that we make. so when we initially launched square, we weren't colorcoding the cells based on the confidence. we added that later on, the kind of like,you know, gray squares and then the low confidence squares. we ask people, "did this help you?"you know. and we measure the difference between those questions as we--as we mix updates.>> thank you. >> hogue: no problem.>> okay. in the early part of your talk, you talked about open source, creative commonscontent. >> hogue: yes.>> and of course, we got google squared, which is google property of course. what's the sortof relationship or current planning to say
create your own sort of corpus that you'reallowed to sort of give back to the community? >> hogue: so...>> if you're allowed to. >> hogue: yes, yes, sure. so freebase is stillopen, right? i mean, i think that's kind of our plan is to keep freebase open in creativecommon license. as i mentioned before, you know, it started of--well, it started verysmall, but when, you know, the summer we acquired it, there was about 13 million concepts. sincethen, we've added a ton of new data through google kind of sources that we've been ableto also creative commons license things like a comprehensive music database with artistsand like, different facts and so on. so that's our plan is to kind of keep that open andthat will be the resource. it also provides
kind of a reference point, if we make anotherpoint here. you know, when your talking about an entity in the world, we want to be the,you know, we want freebase to be the place that you point because it's open so that we'renot--you're not worried about changing. they strive to keep it kind of like wikipedia,very, you know, kind of even-keeled and steady and it has always had identifiers that arekind of unique. so, we would like that to be kind of the way going forward.>> so is there regularity to additional corpus material or is it just whenever your legalteam gets through it? >> hogue: that's a good question. so there'sa couple of answers there. so first, obviously, yes, we need to figure out how to get thedata if we're doing it ourselves. i mean,
that's a big--that's a big problem and we'reworking through that but it's a community effort as well, right? i mean, like we havepassionate users just like wikipedia, then people that are experts in steam locomotivesand they add a whole freebase category for steam locomotives there's an active discussionboard about all that information. that's how freebase has actually grown up until now andi think it's working pretty well. >> thank you.>> hogue: no problem. >> hey, just a question on top of this. anyapis in plan for the dev community to build products on top of google squared?>> hogue: that's a really good question. so, to be completely honest, i haven't thoughtof it yet, so as far as apis and plans, ability
to export a square to a spreadsheet throughgoogle spreadsheet into csb and so on and it will be pretty easy to build an api ontop of. that allows the kind of iterative, you know, querying and so on, and obviously,we have all the tools on google.com search as well. i'd say that most the effort rightnow on the team is focused on bringing what we've learn from squared on to google.comand i think, you know, from there, i think hopefully the tools that we have availableto the dev community on--and then google search, you know, which are constantly being improvedas well, will get the benefit of all the stuff that we learn on squared.>> sure. thank you for that. >> one of the most interesting problems thati seen on google squared is within the set
of attributes. if i understood correctly,you will determine the set of entities from the query and from the set of entities, youtry to find salient attributes on those [indistinct]. >> hogue: yes.>> could you tell us about your ideas to determine attributes from the query? for example ifi ask for arctic explorers, i really don't care when they were born on and when theydied. >> hogue: sure.>> but i'm mostly interested in when they explored antarctica?>> hogue: right. right. so first of all, i think that the approach that we use whichis to first build up a list of actual explorers and then look for their attributes shouldget some of that information, right? because
i think most of the time when you're discussingarctic explorers, those are the kind of attributes that you have. but as well as the fact thati mentioned when we do the initial query "list of arctic explorers" or, you know, "top tenarctic explorers," we look at those tables as well and those pages. so maybe wikipediahas a great table that includes a list of arctic explorers and the dates that they actuallygot to the south pole or whatnot. and so we look at those attributes as well as kind ofanother signal into picking good attributes for the square. with that, i mean, those arethe two sources that we have now which hopefully get you focused. and then the last one, obviously,is user feedback. we'd like to see users add that column to our square.>> so, also that i believe that the query
can be interpreted so that the google squared[indistinct] could cleverly answer cars with [indistinct] by fuel economy.>> hogue: yes, that would be awesome. >> well, will attribute--and so the entityset construction should not take into account fuel economy.>> hogue: right. >> should not eventually take to account thenumber of seats but use it as a post filter. >> yes. no, that would be really awesome.it's definitely something we've thought a lot about. we see these queries on googleas well and obviously, this is something, as i spoke about at the beginning, we wantpeople to be giving us more information in their queries, not less. like, don't tellus the broad thing we want, tell us the specific
thing. so if you're really looking for a sevenpassenger vehicle with a good fuel economy, it would be great if you could tell us that.so one of the things we have obviously in that direction, we haven't used it yet, iswe have a database of attributes so we know what entities they're associated with. so,we can actually look at parsing the query the same way we do with question answeringand say like, fuel economy seems to be an attribute; not, you know, a category. let'ssee what we can do about fuel economy and try to use that for parsing--for picking whichitems to list. but you're right. it's an open area. i think it would be pretty cool.>> thanks. >> hogue: okay.>> google, famously, is strong in terms of
statistical machine learning in the sense,the original google indices were these sort of theoretic models of hubs of authority andthen the entity and value extraction also that you're suggesting also seem like theysmack of parsing and all--[indistinct] in the statistical back-end.>> hogue: yes. >> i'm curious. one of the, as i understandit, benefits of building things semantically, the so-called semantic web, is the notionthat one can attach inference models and logic models underneath a la psych.>> hogue: yes. >> does google have any intention of attachingsystems that, in a sense, would allow sort of reflection. once a set of assertions aremade instead of an entity and values base,
a system could then sort a query itself, reinforceit based on existing corpus of data and draw in new inferences and assertions. do you haveany comment towards that? >> hogue: yes, i mean, i think it's definitelyinteresting a lot of research and i think you're right that the approach that we'retaking because it's so kind of almost anti-ontology, with the exception of freebase, the heart,the good scheme that freebase has, it doesn't--it isn't actually very amenable to it. we'remuch inclined towards building these kinds of broad statistical frameworks as opposedto building, you know, very specific inference engines. i think part of the reason is thatin our experience, we've actually found this to be somewhat brittle. you know, like, youcan only get as many rules as you have time
to write up and we haven't found a good wayof learning them. obviously, you know, there's other domains where that's fine if you havea very specific task that you're trying to accomplish and there's a smaller set even,you know, a thousand [indistinct] or something like that that might accomplish that task,then that's a great domain for that. but we found that, trust me, the types of queriesthat we get, you know, it's just--you have never thought of writing inference rules forthem. so, we've actually--we just found that using the scale to kind of build from thebottom up is a lot easier typically than trying to write rules that go top down. that's justkind of an approach, i think. so i guess my answer is that we don't really have any specific,you know, plans to build these kinds of inference
engines.>> is it all a route of exploration on google's part in terms of its r&d? i mean, there isthe notion that where domains, in essence, tumble out of collections of entities, right?>> hogue: right. right. >> so the notion of cheeses...>> hogue: of course. yes. >> ...can come out that, you know, you maybe able to then talk about the notion of dairy, manufacturers of dairy products.>> hogue: yes, yes. >> so, is there at all any interest [indistinct]?>> hogue: and so, yeah--actually, i mean, i shouldn't--i shouldn't say that we're notworking on it, because i think there's a lot of interest in this. there's a lot of folksin the--in the google research, you know,
kind of area that do think about these thingsa lot and as well as freebase, honestly, a lot of the guys there, because they're sosteeped in the scheme of administering ontology. there's a great language called mql, it'smetaweb query language, that actually allows you in javascript or in json to constructqueries that do very much--you know, would look an awful lot like, in principle--if theysay, "show me all the googlers who have written books published since 2000." and you justkind of express that as a, you know, as a bunch of json that has stars in it and openwildcards. so i think there's a lot of tools for doing that. i guess within my group, it'snot really an approach that we consider. >> thank you.>> hogue: okay
>> do you have any plans for--you suggestthat the more enterprise oriented? >> hogue: so enterprise tools, yes, so withinmy group now, we're mostly focused on doing things for the open web. we obviously do havean enterprise group and, actually, we use a lot of the basic parsers that i was talkingabout before for dates and people's names and so on. and there have been some greatideas there about looking for--looking at those signals to do a better job at rankingof smaller enterprise corpora where you don't have the kind of rich page rank signals andother things like that like you do on the open web. so yes, there's some talk aboutthat. as far as, like, kind of google squared of level, you know, complex analysis, i haven't--ihaven't seen anything like that.
>> okay. going back to something you saidon sentiment analysis. >> hogue: yes.>> at one point, you said that you're using a traditional random field to determine if--whetherin a certain sentiment, if something really is happening and things like that. and youused the word that you were actually training this.>> hogue: yes. >> so my questions is, by training, you meanyou actually have someone tagging a text corpora? you know, like "i had a great time," and thisis, you know, a great phrase? >> hogue: yes, so from the--this is specificallyfor negation detection. >> okay.>> hogue: and as i mentioned, we hand label
a set of 2,000 sentences that had negationsand didn't have negations in them for this part of the sentence that was negated andthen trained on that. so that was a case where we actually went and, you know, spent someactual engineer time to build up a golden corpus rather kind of learning, you know,from the ground up. >> okay.>> hogue: yes. >> and so my follow-up question is kind ofsimilar to what vidal mentioned. it's like, you know, a lot of problems with--you know,a lot of these problems with these sport of sparse [indistinct] with like, you know, forexample, like saying, "i had a great night," is pretty much the same as having--saying,"i had a great evening."
>> hogue: yes.>> and you can determine the--that the evening and night is the same from some other synonymdatabase. i mean, is there any idea of like, you know, training inference engines and stufflike that to actually reduce the sparseness or classify things or stuff like that?>> hogue: sure. yes, it's actually a really good point. and so, yes, we've had a lot ofthoughts down that path. i think there's obviously context as--it's not just the words in thelexicon, the sentiment words that we can use as a distributional similarity metric to identifywhen they're being used in the same context. you can also think about doing it for theaspects, for the nouns, other things like that. so yeah, it's a great idea and i thinkfiguring out that when people say evening
and night, they mean roughly the same thingbecause they're used in the same context, that's one approach. and you can think ofother ones based on, you know, the actual utext and so on. hi.>> hello. good evening. >> hogue: yes.>> i have two questions. >> hogue: sure.>> first one is on google squared. how do you keep track on the back end what peopleare searching for and also to make better searches for that same item in google squared?and then the follow up one to that one is how do you keep track of overseas, how thereare different phrases like when--it's one thing to do within the us or a country thathas, you know, us type slangs. what about,
you know, all around the world?>> yes, it's a good question. so, on the first question which is about how we actually keeptrack of things, we may have logging, right? i mean, we a logs policy where we keep thelogs for a certain amount of time. google squared is nice in terms of privacy in thatwe really don't need much information about a particular user to kind of learning fromthe logs. really, all we really need is their query, so we can strip all kinds of personallyidentifiable information now and just use the raw data. if the person searched for this,then they added this, then they corrected this, you know, that kind of brief sessionis enough information for us to kind of say swiss is a good cheese; people keep addingit when they search for cheeses. so does that
kind of get it? i mean, yeah.>> yeah. >> hogue: yeah. and then as far as the overseas,i think--so first, google square is only launched in english right now. it's pretty tough toget it launched in english and we haven't actually, you know, tackled it in other languagesyet. but in general, the nice thing about the web is that for many countries and manylanguages, there's a really rich set of documents out there already in those languages. in anyof these statistical techniques that i was mentioning where we learned from a large corpusof information, if we can get it to work in english in a general way, it's very prettyto learn it in german or french or something else like that, other places that have differentslang because they have a similarly large
corpus of data as long we could figure outwhich documents are in which languages. does that kind of get at what you're asking about?>> yes. yes. thank you. >> hogue: yes, cool. thanks.>> hello. i was wondering if you guys--like another person asked if you guys were goingto do an api. with a net api, would you--like, do you think would you be able to, like, openup google square algorithms kind of scan other data that's not necessarily on google? like,kind of like--or maybe bundle it up like on a google search appliance type thing?>> hogue: that's an interesting question. so, we certainly could. i think that the problemwith google squared, as i mentioned, it's very reliant on the kind of information thatwe get by doing a google.com search, right?
i mean, it's very reliant on google.com beingtopical, being long tail, you know, understanding lots and lots of information, having goodranking. so if you have another corpus that has similar properties about, you know, havingthat kind of comprehensiveness, yes, i certainly think that that would be possible. as faras the underlying techniques, the ones that aren't based on web search, those are completelyopen, like identifying a date in the document or identifying a fact in the document. thosekinds of things are much more generalized and don't rely as much on web search and asi mention before i think that in the enterprise situation, part of google's search applianceis already people looking into adding those kinds of things.>> yes, so, you know, they'll be able to,
like, search internal blocking and some databasesand stuff? >> hogue: sure. yes, exactly.>> okay. all right, thanks. >> hogue: no problem.>> so you mentioned that there were about 20 million things in freebase, that most ofthem were hand inputs. is that correct? >> hogue: hand is--if by hand you mean likebig script--scraping the site and, you know, kind of reformatting it and dumping it intoa database, then yeah, hand. >> so my question is at what point do youthink that essentially you're going to be able to set out and just sort of scrape theopen web, try to populate more of freebase? >> hogue: yes.>> how far off are we from that sort of thing
and how much human intervention is going tobe necessary to sort of vet algorithms on that?>> hogue: yes, that's a really good question. so i think, as i pointed out, i mean, thedifference between freebase with 20 million entities and the goal of a billion entitiesand, you know, many, many more facts even than that of our kind of fact extraction,tabular extraction of other techniques, i think that we're already--now that, you know,freebase is part of google, we're already looking at how we can take that informationand find the best nuggets of it. the easiest thing to do is just look to augment existingfreebase entities. so, i mean, we probably have more facts about the eiffel tower thanthey do just because the web talks about a
lot more things than perhaps they've beenable to import. so that's an obvious one and it's probably pretty easy to identify thingsthat overlap. and then the harder problem which we've just gotten started on is tryingto identify the new things, you know, and to try to identify whether--it's tough totell whether something is just a different name for a thing or it's actually a completelynew concept you haven't seen before. but yes, so definitely the direction we want to gois taking the kind of script writing and scraping out of the equation and looking much moreat these general techniques which would helpfully scale much further.>> thank you. >> hogue: no problem.>> okay. so at some point you mentioned that
there was sort of a value to doing all theextra work, like a value to each query. how do you figure that out, like, how valuablea query is? >> hogue: that's a good question. i wish ihad an answer that was not kind of poofy. so i was kind of using my own internal metric,right? i mean, if i just need to know when martin luther king was born, the value isprobably somewhat low but the time to execute that query is also low. whereas if i'm tryingto decide where i'm going to college, i mean, that's a, you know, that's 10--you know, a100--$200,000 question these days, right? i mean, like, that's a--that's a pretty bigdeal and so i'm willing to invest an awful lot more time in putting that informationtogether.
>> so does algorithm decide how much processingpower to put into a certain query? >> hogue: yes, again, i wish that we--i wishthat we could do a--priority kind of identify how important a query is but we can't. i mean,i think that that's, you know--i would love if we could figure out an algorithm for that.i can--i can think of ways to start to estimate it. you know, you could look at the amountof clicks and other things like that and how much action did the query get, how long dothe people spend on it, but no. and honestly, we don't actually have a very good metricfor that. i was using more kind of a made up...>> okay... >> hogue: ...theoretical metric.>> do you see this as integrating with advertising
at something?>> hogue: in what way? >> well, i guess advertisers could buy, idon't know, spaces on the grid or...? >> hogue: that's good question. now, thatone i hadn't thought of, honestly. yeah, you could certainly imagine, i mean, having paidplacement, but google squared is completely, you know, ad free right now. but yeah.>> yeah. okay. >> hi. there's been a lot of buzz about core.comand how it's just getting a lot of people using it as a resource. do you think that--likewhat's your take on that site? and do you think it will be a destination resource forsearch and, as they progress, people looking for the right answer?>> hogue: sure. yeah. that's a good question.
i think core is very interesting. i thinkit has some quirks right now. i mean, i think, obviously, it's much better at kind of techquestions. you know, if you go and you're asking about some esoteric--you know, howlong is the olympic track, then you'll probably get an answer obviously but as far as theexisting content on there, it's much better as--telling you how steve jobs is doing perhapsthan other kinds of domains. as far as--i think what they do definitely have is thisability to do two things; first of all, to answer much more complicated queries. i mean,kind of like yahoo! answers and, you know, other things like that and that's somethingthat's been done before. but i think the social aspect is actually really important as welland the idea of kind of trusting the person
that is answering your question and havingsome sort of identity to that. yes, i think that they're doing a great job with that.you know, obviously, aardvark, which we acquired, is also doing things in that domain. but yes,i think that as far as becoming a destination site, i don't know, it's probably at the whimof every other, you know, social startup and whether or not they get enough traction. butyes, they seem to have a good take on it. >> thanks.>> hogue: no problem. hi. >> i was just--i was just wondering is thereanything coming up with google finance because now you've created a huge context, you know?and is there any idea about integrating it to improve google finance people [indistinct]?>> hogue: yeah. that's actually a really good
question. i mean, i think we definitely talkto the team a lot. i think there's a couple of things working against us in finance. but--notto say that we don't want to work on it, but one is that a lot interesting informationis proprietary. it's a little harder to get, so finance tends to do more things like dealsto actually get the data that they want. another thing is that because it's financial information,they have a lower tolerance for bad precision. you know, if you report, you know, the incorrectvalue of some asset or whatnot, you know what, i mean, it's a pretty big deal. whereas ifi tell you that martin luther king jr. was born on 1930 instead of 1929, chances areyou're not going to lose a million dollars over it. so i think that those two thingsare working against us. but you're right,
i think this kind of open extraction to beable to do this kind of analysis over large pieces of information, especially with sentimentanalysis as well, i think is definitely an interesting area.>> so to summarize, it's because it's too big of a risk in market at the moment?>> hogue: to summarize, it's something that we're interested in working on, i guess. i'mnot saying we're not working on it; i'm just--i'm just trying to point out some of issues withactually getting something out the door. >> thank you.>> hogue: yes, no problem. [indistinct[ the room is popular. hi.>> hi. thank you. so given how social the web is and it's becoming in reliance on socialnetworking, is it possible for google squared
to be personalized to people based on theopinions, or beliefs or habits of people in their network friends, things like that?>> sure. yes. i think that's actually a really interesting direction that it could go. ithink some of the things we tried to do when we launched square, we allow you to sharetables. we allow you to kind of edit, you know, see another person's table. we didn'thave a kind of read-write kind of system. it was, you know, a little more complicatedthan we want to tackle for a labs product. but we definitely had this idea that, youknow, i'm planning a vacation; my wife obviously has some say in that, you know, probably moresay. you know, so we want to be able to collaborate on this, where i think that was kind of animportant thing, the idea of saving state
and, you know, kind of iterate through it.as far as getting feedback from your network, i mean, i think to the point about core andother social answer sites, i think it's actually a really interesting area where, you know,it mostly deals a lot with trust. do you trust the site providing information to you? schoolteachers use it too as an example of, you know, how to gauge whether or not--a lot ofour email feedback, you know, is from teachers using this way to teach students to gaugethe trustworthiness of a site and how to actually do deeper research and not just trust everythingyou see because google squared is wrong a lot. that's definitely a good teaching momentthere. but--so we deal with the trust in that way. i think dealing with trust in terms ofthe people that you know are providing the
answers is another great way. it's not somethingthat we've even explored but i think it would be pretty cool.>> searching through their friends on the web?>> hogue: their squares are there. yes, yes, yes, and ranking that. i mean, we do havea social search product that will look at your, you know, your little profile and soon and try to surface results that your friends have shared on reader or whatnot. and obviously,that--those results would be available to squared as part of the ranks documents.>> thank you. >> hogue: you're welcome. hi.>> yes. so if you are dealing with sort of relatively straightforward topics and, say,services at a restaurant or service at a [indistinct],
i think the sentimental analysis should befairly straightforward so you can do very simple machine learning on that. but if you'redealing with much more specific questions and on a certain topic of, say, a [indistinct]and engines, how do you handle that, i mean, to pick out just much more specific things?do you have any ideas of how to handle that? >> hogue: so just to make sure i understandwhat you're--so you're asking about the difference between kind of a broad domain with lots ofevidence like local businesses versus the narrow domain.>> yes. yes. >> hogue: like, i don't know, a car--autorepair or something else, i think, right? >> you know, specific part of an engine, youknow.
>> hogue: do people express strong opinionsabout specific parts of engines is one question? yes?>> no, but it could be valuable for a lot of people if you are deciding what to do about,you know? >> hogue: sure. i mean, i think some thingsare universal, right? i mean, like, there's--most sentiment terms, kind of, you know, the waythat people express frustration or happiness tend to apply to all domains, and it's thenouns and the aspects that change out for [indistinct]. they'll maybe be referring tothe carburetor in one domain or the wait staff in another domain. so i think the nice thingabout the way that we approach sentiment analysis is that lexicon that tells us what's positiveand what's negative should be universal once
we've developed it. and then it would justbe a matter of--i mean, i don't know if there's different words that you might use to expresshappiness... >> but i'm thinking if this is an engine,i mean, like, maybe positive is very specific, what's the core temperature range or like,you have like, say, like your peak--your... >> yes, like peak torque or something likethat is in a certain range and [indistinct] really happy about that.>> yes. >> hogue: yes, yes. yes, it's really interesting.i think we--i can imagine using some of the tools that we have to build something likethat but, as i said, most of our work is focused on the broadest possible applications. soyes, i think we have the tools to build that.
it's not necessary an active direction thatwe're going right now though. >> thanks.[indistinct]>> hogue: no problem. hi. >> hi. do you ever think about incorporatingvideos and their content into google squared? >> yes.>> like, you know, if i wanted to look up fitness videos or something like that.>> hogue: yes. actually, the first time we showed google squared before released it toour vice-president here in new york, steve feldman, we led off with the query of rollercoasters which was kind of cool because we can show the heights and the top speeds andall that. and his suggestion was to add videos. he's like, "wouldn't it be cool if you could,like, watch a video of each coaster going
down its main drop?">> people vomiting. >> hogue: yes, exactly. so, yes, i think totally.i mean, i think the main issue there will be getting a video that's on topic, right?i mean, like, we--a lot--we still struggle with square. there's a roller coaster calledmantis and it's really tough for us to get a picture of the roller coaster and not apicture of the bug. so i think video has even fewer signals would be my intuition, and we'dhave to work a little bit to make sure they're on topic but i think we could probably doit. >> thank you.>> hogue: yes? hi. >> hey, a little bit, i've discussed withyou earlier but just to give the background
that similar application have been workingbased on social networking and one thing i just wanted to check with you; how did youguys improve the intelligence from 50 to 60 or the satisfaction? because we found, workingon those, improving it manually very specific to particular queries was pretty tedious joband not every time relevant because it's a time sensitive query. some body's lookingfor game on a particular day and improving that query won't make sense.>> hogue: right. so we try very, very hard--and you probably got the gist during the talk--notto do this kind of manual one off bespoke kind of fixes, right? we have a little bitof white listing and black listing. we try to avoid, you know, racial slurs and thingslike that and it's--you know, we want to make
sure to get those things right. but in general,we're really trying to do the most general possible thing. so you asked about how wewent from 50 to 60%, we've actually gone further than that since then. it's through lookingat the broadest class of errors that we might make and attacking it in the broadest possibleway. so, for instance, maybe--i don't know the specific, you know, improvement one atthat point, but we did things like add a new general extractor. you know, we added somethingthat looked--maybe we were only looking at tables and now we're going to look at thoseclear problems, you know, attribute colon value and try to find more data that supportsthat. we might look at a new signal that's the next, like, user refinements on google.comthat might give us more of a signal as to
what's relevant and what's not. so typically,when we make those kinds of improvements, we try to look at a very broad case and wetry to do any manual fixes, with the exception of these kinds of, like, worst case scenariokinds of blacklist and things like that. and as far as the [indistinct] issue, you know,there's certainly things that we get wrong. question-answering, amit singhal, our googlefellow for search submitted that we were getting the--i think it was the prime minister ofindia or something like that in your list today, the arrow we had. anyway, when we'regetting some question wrong, he sent it to us and it was--it turned out that he had justgotten reelected, and the new table that we're pushing is going to fix that. so we'll fixthings kind of only in a general way but time
sensitive stuff is still something we strugglewith. >> sure.>> hogue: yes. >> all right, thank you.>> hogue: yes. no problem. hi. >> hi. how would google squared deal withlife-critical or mission critical questions like my dog just swallowed rat poisoning;what should i do? >> hogue: google squared wouldn't deal withit at all, i hope. google squared is mostly about categorical queries. so i mean, like,maybe it would answer ways to kill my dog or something like that but maybe not. but,you know, google.com is actually probably a much better source of information for thatand there, i think, we're going to try to
aggregate them [indistinct] into the web.hopefully, it would go to answers. cora i think would be a great site probably. in theory,it would be a great site to answer a question like that, but it's not really a domain forsquared. yes. >> could you actually turn it away if theyhad those kinds of questions or...? >> hogue: no. it's an interesting point. wemight want to. she had a good question with whether we should turn users away if theyask something like that. one problem is that it's hard to identify those. you know, as,you know, as we were talking about before, like classifying queries like that can bedifficult. maybe if there's certain very, very obvious buzzwords, you could build aclassifier. these kinds of things are very
tough. you want them to be 100% percent preciseand 100% covered and that just doesn't exist in these kinds of domains. yes. my hope wouldbe that a user would never find them self on google squared for a question like that,to be completely honest. yes? hi. >> going forward, how do you see google squaredintegrating with mainstream web search? or i guess in other words, if you see that auser has searched for something that could be better represented by google squared, doyou ever see--do you ever think you would see a squared page showing up in place ofurls? >> hogue: yes. i mean, it's certainly somethingthat we would love to do. as i mentioned, we we're taking parts of squared that we believecould be really useful like the question answering
bit and we've already launched that and we'repushing that on google.com. we're taking other pieces in that they're kind of in active developmentas--they have to be kind of restructured and rethought because they don't quite make sensein exactly the same way. but we think that they're very valuable to answer these kindsof queries. as far as google squared knowing enough to kind of take over the whole pagewhen you type in, you know, hybrid suv, it's a little bit of a stretch but i can certainlyimagine at least being kind of--there being a way to get it immediately from your searchresults. but google is mostly about getting you to the destination of where you can bestserve the information, so. >> and do you have any figures on how manygoogle web searchers could be represented
by squared? is it--is it common to have searchescome in that you would say, you know, let's show this in the grid?>> hogue: that's good question. i don't have current figures. i do know that they're fairlyfrequent. you know, i don't necessarily want to say exactly but, you know, they're--youknow, they're not--it's a non-trivial portion of our query stream. hi.>> you said the users [indistinct] is like allowing with the precision and you can measurethe users [indistinct] by, like, doing a survey. but how do you--how do you measure own precision.>> hogue: yes, so measuring our own precision, this is something we do kind of a cross searchquality. basically, it's a--we build up a set of queries that we care about, we buildup a set of answers we believe are correct,
and then we kind of measure it. we automaticallyquery our system and check whether or not we're getting things right and we measureprecision of that and we call up that--just like, you know, any other kind of machinelearning task. >> so that's like way existing before.>> hogue: i'm sorry? >> that's like existing way before the googlesquared. >> hogue: sure. yes, yes, we do this for alltypes of quality problems. >> and the other question, is google squareis like a good tool maybe for academic research? >> hogue: yes. in fact, as i said, as i mentioned,that's one of our key demographics is school teachers and so on. but yes, i mean, thatwas definitely the book report or, you know,
kind of research report domain is one thatwe sell at. >> yes, because i see that you have imagelike embedded in the [indistinct] sheet. what about the other maybe pdfs or those [indistinct].>> hogue: sure. yes, i mean, as i said, if we can get people the information that wouldhelp them, we do pull facts from pdfs because they're part of the google index but surfacingthem, yes, might be that interesting route. >> thank you.
Post a Comment for "hybrid suv edmunds"