ELWsoftware.com

Multiobjective optimization means simultaneously finding a good solution that works for multiple measures of success.  Is Google currently using this idea to determine page rankings?  Can ELW reverse engineer Google’s search algorithms by assuming they are using these techniques?  The trick would be to figure out what Google wants to improve.  Obviously, Google makes money when searchers click on ads next to search results.  Directly optimizing search results to increase revenue would probably lead to crappy pages taking up the first few positions.  This would increase the number of people clicking on ads instead of search results.  I’m not so sure Google would do this, since the extra revenue may be at the cost of losing those customers next time.  The only alternative left is for Google to try to maximize long term revenue by maintaining the quality of organic search results.  This is a multiobjective optimization problem!

First, let me say that this is only speculative.  Anyone who has read my other articles can figure out what my basic view on Google are.  I suspect that what Google does is much less sophisticated than any theoretical optimization problem would allow.  Good optimization techniques are slow, and don’t scale well to anything near the size of Google.  It sure is lucky for the world of computer scientists that simple and fast ideas work so well.  Sometimes “dumb” solutions actually work better!

Forget everything you think goes into a search engine’s ranking algorithm for a minute.  How would you compare the quality of two different algorithms?  You need to actually put them into practice and look at the click-through data!  Which ranking components lead to the most clicks on the first few results?  Which bad ranking components lead people to enter a different query, or look at search results on the second and third pages before they find what they want?

I have done quite a bit of experimenting on this, and have developed a combined algorithm for improving search engine ranking algorithm quality by optimizing for positive click feedback.  Contact ELW if you have a few million dollars and want to give Google a serious run for their money!

This morning Google turned off some of the less used mechanisms for obtaining search results.  The search result scraper Scroogle.com is temporarily unable to provide search results.  The site used to be a useful tool to get search results without any warping due to location and user info.  A quick check of the SEO tool Rank Checker shows that search engine rankings are not provided for the Google column anymore.  There is no possibility that this is a coincidence.

I can only speak for ELW, but I hope that the world sees this for what it is.  Google only wants you to use their services when they control the medium.  This makes some sense if they were losing money from scrapers, but we know that isn’t true.  This truly is an act of aggression against professional SEO firms everywhere.  Don’t get me wrong, ELW has not been inconvenienced at all because our research doesn’t rely on the same back doors as Scroogle and rank checker.  Maybe this is a good thing, and no complaints are necessary?

There is much more to this outage than meets the eye.  I’ll save the real conspiracy theories for our paying customers!  More to come.

SEO blogs including this one often mention that search engine ranking algorithms at major search engines are kept secret.  The reason given is that knowing knowing the exact algorithm would allow websites to manipulate their rankings too easily.  I don’t think this is exactly true.  If a search engine admitted that it only used Google’s Pagerank, it would still be extraordinarily difficult to manipulate search results.  The real reason for keeping the algorithms a secret has nothing to do with websites wanting higher ranks.  A search ranking algorithm made available to the public could almost instantly make a search engine obsolete.  If you knew the search algorithm used by Google, you could reproduce the majority of their search functionality with a modest investment in hardware.

If figuring out how Google works is so easy, why hasn’t anyone done it?  I would argue that it has actually been done already.  The real difficulty in reproducing Google does not lie in figuring out the ranking algorithm.  That would be both useless, and expensive.  If you want to reproduce Google’s search capability you need to spend money.  Google is able to spend billions of dollars on its infrastructure because people give them billions in ad revenue.  This is the real secret to reproducing a good search engine.

Wouldn’t it be fun to reverse engineer Google’s search algorithm for SEO?

1. Create a spider program to analyze and index websites.
2. Create a ranking algorithm to sort them according to some measure of relevance.
3. Repeat!

Oh, did you think it would be more complicated than that?  The biggest myth in SEO is that search engines are complicated. They aren’t.  What is complicated is storing a cached copy of every website on the internet.  Add to that the difficulty of running a database capable of generating search results to everybody on the planet at the same time.  Now you see the real secret:  There are no important secrets!

If you want to build a better search engine, the problem is to change what is meant by search.  I think the future holds a better solution than what we have today.  Returning a list of pages is not a very modern way to search.  The only reason Google still does it that way is because they found a way to make money selling ads next to search results.  Organic search results are just there to sell advertising.  It would be an interesting project to create an open source search engine, and ELW is already on the case.  Stay tuned for regular updates.

Google is up to its old tricks again.  It wasn’t enough that they store every search query you make, and use your browsing habits to sell advertising.  There are still some websites out there that don’t use Google Analytics or Adsense to track your movements!  Google now wants to experimentally test a fiber optic high speed internet service in a few communities.  This would be just like the holiday gift of free wifi in airports they gave us.

This plan would allow them to track everything you visit on the internet, not just the sites using Google tracking.  It that what we want Google to be doing?  Is your ISP so terrible that this is the choice we are now forced to make?  It hasn’t even been a week since anonymous employees at Google leaked that the U.S. National Security Agency has been infiltrating them!

Google is a cool company, and they do some cool things, but they can’t be trusted!  This has nothing to do with how fast the internet service is that they are offering.  I know that every ISP out there is terrible.  Google will be just as bad!  Google can’t sprinkle magic dust on everything and make money out of thin air.  Having faster internet service doesn’t translate into greater revenue for Google.  That is unless your tax dollars are going to be funding this experiment through the budget for our intelligence services!

Doesn’t the whole thing make a crazy conspiracy theory look likely?

1. There is no REAL danger of terrorism, but they tricked everyone into paying for more intelligence capabilities.
2. The cheapest way to gather information about people is to trick them into voluntarily giving it to you.
3. The NSA can’t make a search engine because nobody would trust them.
4. Oh, cool!  Google already has a search engine that stores everything about everybody?
5. NSA is confirmed to be working with Google.  “Don’t worry, they’re just helping with cyber-terrorism”
6. Don’t worry!  Google says they aren’t evil.

I’ll leave it up to the readers to connect the dots on this one.

A short time ago I wrote an article confessing my suspicions about Google being too similar to Enron.  When you are confronted with a situation where not everything seems right, you should start checking facts.  Just start with something simple, and see what you really find out.  Enron didn’t steal for so long because they were such good thieves!  Enron stole for so long because nobody asked them the hard questions about where they were making money.  Everything I hear about Madoff is the same story: Even though it looked too good to be true, nobody checked.

Well, does Google’s revenue look too good to be true?

• Google reported 6.5 billion dollars in revenue in the most recent quarter from advertising alone.
• Google reports that 66% of that revenue was from advertising on the Google website.  The rest was from Google network sites (probably adsense type systems).  I mention this because regular site owners can earn money by placing ads from Google on their pages.  Naturally, there is some motivation for some site owners to click their own ads for their own benefit.  The 66% of revenue on Google sites all goes to Google, so there shouldn’t be any fake clicks.
• This leaves around 4.4 billion dollars in revenue in 3 months from people clicking on ads on Google’s pages.  Google uses pay-per-click ads so the money only comes from users clicking ads.  Nobody really pays for their ad to be seen and not clicked.
• The price of a single click on a Google ad can go for as little as 5 cents, and sometimes a bit over $50.  The $50 clicks are usually for local lawyers and don’t get clicked very often.  Most likely the average price per click is below one dollar, but I don’t have the statistics.  I only have my own experience from my own Google ad campaigns, and those of my clients.  Conservatively, I would put the average cost of a click a bit higher because the topmost ads cost more, and are clicked most often.  How does two dollars sound as an estimate?
• At an average of $2 per click, this would mean that Google managed to get over two billion ad clicks in 3 months!

I know people buy a ton of things on the internet, and that some users just love using Google.  From my own experience I know that using Google’s Adwords ads is an excellent way to find customers who are ready to buy your products.  When I explain how Google’s advertising works to ordinary people, I always get a similar response:

But nobody clicks on those ads!

It is true, I almost never click on ads when I am doing a Google search.  In fact, my experience with Google ads shows that the click-through-rate on a Google ad is rarely above 2%!  In most practical cases the rates are much lower, since there are often ads on multiple pages where hardly anybody clicks.

Look at the English-speaking world population for a while.  No, look at the English-speaking population of the world with access to a computer, who can afford to buy a product online.  Is it really a big enough group of people to produce 2 billion clicks on Google ads in 3 months?  I realize that a long tail phenomenon is at work, and that there are probably people who click a whole bunch of ads.  This would also mean that there are hundreds of millions of people who never click an ad!

Let’s start asking real questions:

• Does the revenue earned by Google make sense, given what we can extrapolate about the population?
• Does the possibility exist that Google has fudged the numbers?
• If Google was making money from some other source, would it be difficult to disguise the source by saying the money came from ad clicks?
• If Google was selling your personal information to corporations or intelligence agencies, would it be possible for them to pretend it was advertising revenue?

Google has only one real source of income, and that is advertising.  The only way to believe Google is telling the whole truth about their income is to believe that all the money they earn eventually started from regular businesses paying for their ads to show up on Google.  Since the vast majority of people I ask never click ads, I have to assume that most of the ad clicking is done by a relatively small number of people.  Since nearly all the businesses I talk to have never even heard of Adwords, I have to assume that the ad buying is done by a relatively small number of businesses.

Are these two groups of ad-clickers and ad-buyers really enough to generate 4.4 billion dollars in 3 months?

I think search engine optimizers have become a bit lazy recently.  Google has been taking it easy too, and collecting their money.  SEO is a game, and the players are you and Google.  Google is going to remind their real competitors, you and your SEO, what makes a good search engine work.  Let me summarize the current view of SEO that is about to change:

1. Google started using inbound links and pagerank to get rid of spam websites.  Back years ago it was possible to trick search engines and rank higher than Microsoft, for example, by putting “Microsoft” a thousand times in text the same color as the background.  Google’s logic was that it was more difficult to fake inbound links than text.  They were right.
2. Since it really is profitable to be ranked higher in search results, people never stopped trying to achieve better search engine rankings.  With Google, this just meant getting inbound links then became the most important SEO trick.
3. Cottage industries appeared (SEO), and offered services meant to improve your search engine rankings.  And the ones who aren’t outright scams actually do some real work.  Most people offering SEO services are able to turn an absolute crap website into a crap website with a slightly improved search engine ranking.

Hasn’t anybody noticed the change in what SEO tactics work recently?  Is your SEO company even paying attention?

Most SEO articles I read talk about Google as though it is running the same algorithm as it did 10 years ago!  They aren’t, and it could cost you quite a bit.  Here is the deal:  I have confirmed my own suspicions by asking some anonymous persons who know, and Google has changed their system in 2010.  Here is what I suspect according to the research lab at ELW.

1. Even very “small” websites can easily get ranked on the first page for some pretty good keywords with zero inbound links from other sites!  There is evidence that this can happen within hours of posting.
2. Paying someone to generate a ton of content for you all at once will not work, even if it is from all different domains linking to you.
3. Generating a thousand one-paragraph articles on your site all at once is not an effective numbers game that will pull in “long-tail” queries.
4. Don’t hire foreign writers to generate content for your English language site! It is an absolute fact that computer science has given us the tools to decipher the origins of text.
5. Don’t even think about using computer generated articles! It is so easy for a computer to detect this junk.  Even the ads for this software sounds foolish with claims like “With this software I can crank out 5 articles a day.”  Five articles a day?  If you can only produce five articles a day, and software does all the work for you, you need to be back in high school!  I won’t provide a link or name for the ad, since I don’t want anybody to buy their stuff.  It’s even possible that this article spawned one of their adsense ads on my page!

Seriously folks, I think Google has figured out how to rid their search rankings of most of the SEOed spam out there.  I usually don’t like to give Google too much credit, but they have definitely made some improvements.  Took their sweet-ass time though, didn’t they?  Now we’re even.

Take a look at the present situation and realize that Google has become a giant company.  That fact alone doesn’t scare me at all.  There have been and there will be bigger companies out there.  If you want to compete with Google you need to have a plan, and we have a few ideas!

• First we need to recognize that Google does not have a monopoly, just a very dominant market-share.  I think Google themselves would be the first to acknowledge that making a competing search engine is possible.  They probably welcome competition.
• Second, we need to pay careful attention to the profit situation.  Google didn’t become mega-rich because its search engine is so awesome, they did it by selling ads!  Adwords allows ordinary people to rent a link to themselves on search results pages for any search phrases they choose.  You only pay when somebody actually clicks the link!  This idea is what Google did differently, and changed the game.
• Third, recognize the problems with Google.  Sometimes it’s hard to find flaws with existing products.  Don’t think about what is wrong with the search process; instead think about what could be better!  It truly is the little things that make a big difference.  Think creatively!  I give a few examples from other web sites.  Amazon’s recommendations often think you want to buy a “blue” version of something because you bought a “red” version.  Maybe this works for them, but it does make me question their recommendations.  Ebay doesn’t allow you to only look at negative or neutral feedback.  You have to go through page after page of meaningless “great ebayer!” to actually find the truth.  These are problems you can only see if you compare what exists with how things could be.
• I’m sure there are thousands of computer scientists out there that could easily have implemented the search algorithms necessary to produce a competitor to Google.  When some idea finally comes along to compete with Google, it will also be easy to program.  Sometimes a project like this just needs to get a little push in the right direction, like Linux.  Linux didn’t get created to compete with Windows.  Someone just decided one day that they wanted a free Unix operating system.

Well, today I am DECIDING to compete with Google.

Post any ideas please.  I want this to be a collaborative thing with everyone out there.

The pagerank system used by Google is a major factor used to determine the quality of a page, and correlates well with how it is positioned in search engine results.  The pagerank number is supposed to represent a measure of the probability that a person randomly clicking links on pages will land on your page.  There are a few enhancements to pagerank that prevent dead end pages, and loops of pages from getting infinite pageranks.

The pagerank calculation is designed so that the average pagerank of all the pages on the web is 1.  Achieving higher pageranks is accomplished by having inbound links from other pages.  Pages with higher pageranks pass on some of their value through these links.

The obvious question is whether you can increase the pagerank of a site by creating more pages with links to it.  The obvious answer is YES.  The problem is neither Google nor any other modern search engine uses the pagerank algorithm as it is intended.  This would be too easy for spammers and anyone else to fool.  You could just make a million pages and link them to any page you want.

The actual algorithms used by major search engines consider many other factors:

1. Are the incoming links all from the same page?  The same site?  The same IP address?  A good search engine can tell the difference.
2. Is the content all from similar pages?  A million pages linking to your site would be nice, but not if they all have the same content.  Doing this might get your site banned from the search rankings.
3. Not only is the content used to determine whether pages are different, it is used to help match up with the search phrase.  If all the pages linking to yours have text related to yours, this will help your page.  This is how a good search engine works to prevent a major site from dominating the search results on unrelated queries.
4. It has been confirmed that user behavior affects search rankings.  This means that major search engines are no longer viewing pagerank and similar algorithms as an approximation of how a randomly-clicking user navigates.  They are more likely using actual behavior instead of random behavior.
5. Link location and visibility matters to varying degrees.  A prominent link at the top of a page, or within an article, counts for more than a link hidden by making its color the same as the background.  This also means that a link at the bottom of a long page is not as good as one at the top.  Major search engines have done a great deal to make their algorithms correlate with what a user actually sees and clicks.
6. If your page visits only last for a split second before the user hits the back button on their browser, this has an effect on ranking.  For a link that is followed by a user who quickly backs out, the algorithms may view this as a mistake by the user.

In summary: Adding new pages does increase Pagerank, according to the Google patent.  The problem is that nobody, including Google, uses the actual pagerank without modifications.  Focusing on increasing Pagerank may help you, but it is better to focus on generating new content that people will want to read.

I have been talking for years about how Google is the most Enron-like company out there.  Now I find out that Google has plans for trading energy!  A quick overview of my Google/Enron arguments:

• Adwords! Google makes nearly all of their money through a process that is only completely transparent to themselves.  Adwords advertising is what makes Google its money.  Everything else is small change.  It takes a lot of $1 clicks to turn into billions of dollars a year, but everybody I talk to says they “never click them” or “nobody clicks those ads!”
• The smartest guys in the room! This documentary film about Enron is starting to seem eerily familiar when I watch Google.  Hire the smartest people you can find, so it gives the illusion that you are really doing something.  Now, if anybody questions how you are able to make so much money, you can just say they are stupid because they don’t understand!
• The Black Swan! This phenomenon refers to the idea that because all the swans you see are white, black swans must be impossible.  A company grows from nothing into the the most dominant force on the internet in just a few years.  It must be perfect!  Its stock will never go down!  They have never made a mistake, so they never will!  Hooray?  Be careful.

I understand that Google probably uses more electricity than anybody other than the national security agency, so it makes sense that they would want to get more involved with energy trading.  Google makes a lot of money, so they should be involved with banking too right?

The problem is that the public trusts Google too much.

Companies that make massive entries into new and unrelated businesses often fail spectacularly.  When you get a bunch of smart people together to make money, the last thing you should do is trust them.  Do you want the information Google stores about you influencing your power bill?  Do you want to upgrade your thermostat to a cool new one from Google?  What if it provides you with free wifi?  Yes I’m joking, but Google isn’t.

After traveling during the holidays I noticed that half of the airports I visited had free wifi as a holiday gift from Google (It probably would have been more, but some had their own free wifi already).  Am I supposed to believe that major corporations give holiday gifts just to be nice?  Of course not!  If the NSA or CIA decided to give everyone in the country free cell phone service as a holiday gift would you believe them too?

Never believe the REASON given for something being free!

The obvious conclusion is that Google found a way to make money from giving free wifi in airports.  Remember, with Google, You are the product! Google has never made it a secret that they store every bit of information they gather about you forever.  This is because they sell this information to advertisers in the form of adwords ads.  The advertisers who want customers of a certain type pay to get their ads shown to them.  The idea is that the ads are only shown to the most likely customers.

1. Google gives away free software and services as bait because they want you to give them information.
2. Google’s search algorithms are not optimized for relevance, but to increase their adwords revenue.  This just happens to correlate very well with their definitions of relevance, in case you are paying attention.
3. If you don’t use the Google search engine, they still keep track of your web habits through sites that use Google Analytics, which includes a significant portion of the web.  Don’t believe me?  Download the noscript plug in for Firefox and see how many sites use it.  Better yet, view the page source in any browser.
4. If Google is providing wifi, they know everything about your internet use.  Probably more than everything actually.  Obviously they know what airport you are at.  They know when you are there, and probably your general location in your terminal.  Add it all up and they probably can figure out what flight you are taking.  This means that Google has gone beyond using simple keywords to target ads to you!  Now instead of knowing that you searched for hotels near Chicago for example, they know when you will get there.  They know if you are taking an expensive airline.  They know if your flight is delayed, meaning you are probably pissed off and tired.

Seriously, be careful!  Google has never argued the fact that they collect data about you.  They say that your privacy is safe because if they screw up, you won’t trust them anymore.  Is there really a difference between information obtained from your computer by a virus, versus given away to Google?  It is the same information after all.  It’s not like Google is spending much money protecting your privacy.  They just want to make as much money as they can until the inevitable problems start to show up.  Sortof like Enron, right?  They know that eventually there will be a major security breach, they just don’t know when.  When I say major security breach, don’t underestimate what I mean.  A bank losing a few thousand credit card or social security numbers is nothing compared with what Google has on you.

• Expect Google to start providing wifi in more and more places.
• Expect adwords, and the use of keywords to disappear as the main method for generating revenue.
• Expect the new Google phones to be the most dangerous piece of technology ever released to the public, so far.
• Expect some sort of major money-laundering issues to come up.  Possibly in 2010.
• Expect to forget what Google “used to be like” and get used to the new Google that regularly sells information about you at a scale that would have shocked George Orwell.
• Expect that Google will patiently and calmly hold the world and its economies for ransom.
• Last one:  EXPECT THAT NOBODY WILL EVEN PAY ATTENTION!