ELWsoftware.com

It seems like every few weeks now the Google scraping site Scroogle completely falls apart.  Last time it happened they blamed Google for shutting down some old link to search results that they were using because it was easier to scrape.  This time they are blaming Google again, even suggesting that scroogle may be shut down permanently.

Any regular reader to this site can infer that I don’t like Google for a number of reasons, the most important being their privacy practices.  Every search query, and what you click on is stored by Google.  Wouldn’t this information be a goldmine for modern paranoid intelligence agencies?  It is.  I would love to dedicate an entire article about the specific instances of government agencies viewing certain “profiling” articles I have written.  When someone uses Google, and clicks on your site in the results, your site can see the query used to find yourself.  This is just the tip of the iceberg compared with what Google has.

So, Scroogle and sites like it have value. Don’t blame Google for your problems keeping your site operational.  Be real computer scientists and write some new scraping code for crying out loud!  Get someone else to do it for you otherwise.  Why not get some real street credibility and write your own search engine?  Seriously, there are undergraduate programs all over the country that do this as a project in semester-length courses.

The Scroogle scraped results are not available.  I’ll bet there are at least 100 computer scientists out there reading this article that could solve this problem over the weekend.  In the meantime, throw together some results from other search engines.  Scrahoo!  Scring!

I want to go on the record and say that ELWsoftware.com will solve this Scroogle problem for the low low price of $10k.  I know it can be done for less money, but I don’t hear anyone else offering.

Are we really in a pre-historic age when it comes to search engines?  No, but we have barely begun to use the real power of computers to organize the information we seek.  This will be less of an article and more of string of thoughts.  What can Netflix teach us about how search can be improved?

• How often do you find what you were looking for using Google?
• How often do you search for something when you don’t know what you are looking for?
• It is much harder to find something when you don’t know what it is called.  This is what is wrong with mathematics:  Don’t name ideas after their creator/discoverer.
• Don’t organize any information solely on names or keywords!
• Netflix would be a terrible service if all they had was lists sorted by title, director, and actors.
• Netflix uses collaborative filtering to find you movies you might like.
• Collaborative filtering works even when you don’t know what you are looking for.
• Why don’t they use collaborative filtering for search engines?
• There are just too many web pages out there to make it practical.
• Does collaborative filtering work for books?  Yes, Amazon uses it.
• Can the Netflix method be used to produce search results without even searching?  Yes.
• Has anybody built a collaborative filtering search engine for the web?

Google is up to its old tricks again.  It wasn’t enough that they store every search query you make, and use your browsing habits to sell advertising.  There are still some websites out there that don’t use Google Analytics or Adsense to track your movements!  Google now wants to experimentally test a fiber optic high speed internet service in a few communities.  This would be just like the holiday gift of free wifi in airports they gave us.

This plan would allow them to track everything you visit on the internet, not just the sites using Google tracking.  It that what we want Google to be doing?  Is your ISP so terrible that this is the choice we are now forced to make?  It hasn’t even been a week since anonymous employees at Google leaked that the U.S. National Security Agency has been infiltrating them!

Google is a cool company, and they do some cool things, but they can’t be trusted!  This has nothing to do with how fast the internet service is that they are offering.  I know that every ISP out there is terrible.  Google will be just as bad!  Google can’t sprinkle magic dust on everything and make money out of thin air.  Having faster internet service doesn’t translate into greater revenue for Google.  That is unless your tax dollars are going to be funding this experiment through the budget for our intelligence services!

Doesn’t the whole thing make a crazy conspiracy theory look likely?

1. There is no REAL danger of terrorism, but they tricked everyone into paying for more intelligence capabilities.
2. The cheapest way to gather information about people is to trick them into voluntarily giving it to you.
3. The NSA can’t make a search engine because nobody would trust them.
4. Oh, cool!  Google already has a search engine that stores everything about everybody?
5. NSA is confirmed to be working with Google.  “Don’t worry, they’re just helping with cyber-terrorism”
6. Don’t worry!  Google says they aren’t evil.

I’ll leave it up to the readers to connect the dots on this one.

I have spent all day trying to get my legally purchased copy of Dreamweaver working on my computer.  It used to work fine, but now it crashes before it finishes opening!  I have plenty of memory (12GB) so that is not the problem.  (I don’t even know why Adobe bothers to ask).  Everything else in CS4 works, including photoshop, illustrator, and fireworks.

I have heard and tried many different fixes and so far nothing works:

1. Uninstall and reinstall Dreamweaver.  No change.
2. Delete configuration files.  No change.
3. Tried opening from a different user login.  No change.
4. Repair disk permissions.  No change.
5. Somebody suggested that the problem was caused by daylight savings time.  Nope.  The time was before, and is now correct.  No change.
6. Uninstall Dreamweaver, delete all dreamweaver files, reinstall Dreamweaver.  Install completes without errors.  No change.

I have never had so much trouble with any software that I paid for.  Oracle was worse, but that was a free download.  Seriously Adobe!  What the hell is going on?  Did you hire a bunch of programmers from Microsoft or something?

A short time ago I wrote an article confessing my suspicions about Google being too similar to Enron.  When you are confronted with a situation where not everything seems right, you should start checking facts.  Just start with something simple, and see what you really find out.  Enron didn’t steal for so long because they were such good thieves!  Enron stole for so long because nobody asked them the hard questions about where they were making money.  Everything I hear about Madoff is the same story: Even though it looked too good to be true, nobody checked.

Well, does Google’s revenue look too good to be true?

• Google reported 6.5 billion dollars in revenue in the most recent quarter from advertising alone.
• Google reports that 66% of that revenue was from advertising on the Google website.  The rest was from Google network sites (probably adsense type systems).  I mention this because regular site owners can earn money by placing ads from Google on their pages.  Naturally, there is some motivation for some site owners to click their own ads for their own benefit.  The 66% of revenue on Google sites all goes to Google, so there shouldn’t be any fake clicks.
• This leaves around 4.4 billion dollars in revenue in 3 months from people clicking on ads on Google’s pages.  Google uses pay-per-click ads so the money only comes from users clicking ads.  Nobody really pays for their ad to be seen and not clicked.
• The price of a single click on a Google ad can go for as little as 5 cents, and sometimes a bit over $50.  The $50 clicks are usually for local lawyers and don’t get clicked very often.  Most likely the average price per click is below one dollar, but I don’t have the statistics.  I only have my own experience from my own Google ad campaigns, and those of my clients.  Conservatively, I would put the average cost of a click a bit higher because the topmost ads cost more, and are clicked most often.  How does two dollars sound as an estimate?
• At an average of $2 per click, this would mean that Google managed to get over two billion ad clicks in 3 months!

I know people buy a ton of things on the internet, and that some users just love using Google.  From my own experience I know that using Google’s Adwords ads is an excellent way to find customers who are ready to buy your products.  When I explain how Google’s advertising works to ordinary people, I always get a similar response:

But nobody clicks on those ads!

It is true, I almost never click on ads when I am doing a Google search.  In fact, my experience with Google ads shows that the click-through-rate on a Google ad is rarely above 2%!  In most practical cases the rates are much lower, since there are often ads on multiple pages where hardly anybody clicks.

Look at the English-speaking world population for a while.  No, look at the English-speaking population of the world with access to a computer, who can afford to buy a product online.  Is it really a big enough group of people to produce 2 billion clicks on Google ads in 3 months?  I realize that a long tail phenomenon is at work, and that there are probably people who click a whole bunch of ads.  This would also mean that there are hundreds of millions of people who never click an ad!

Let’s start asking real questions:

• Does the revenue earned by Google make sense, given what we can extrapolate about the population?
• Does the possibility exist that Google has fudged the numbers?
• If Google was making money from some other source, would it be difficult to disguise the source by saying the money came from ad clicks?
• If Google was selling your personal information to corporations or intelligence agencies, would it be possible for them to pretend it was advertising revenue?

Google has only one real source of income, and that is advertising.  The only way to believe Google is telling the whole truth about their income is to believe that all the money they earn eventually started from regular businesses paying for their ads to show up on Google.  Since the vast majority of people I ask never click ads, I have to assume that most of the ad clicking is done by a relatively small number of people.  Since nearly all the businesses I talk to have never even heard of Adwords, I have to assume that the ad buying is done by a relatively small number of businesses.

Are these two groups of ad-clickers and ad-buyers really enough to generate 4.4 billion dollars in 3 months?

Take a look at the present situation and realize that Google has become a giant company.  That fact alone doesn’t scare me at all.  There have been and there will be bigger companies out there.  If you want to compete with Google you need to have a plan, and we have a few ideas!

• First we need to recognize that Google does not have a monopoly, just a very dominant market-share.  I think Google themselves would be the first to acknowledge that making a competing search engine is possible.  They probably welcome competition.
• Second, we need to pay careful attention to the profit situation.  Google didn’t become mega-rich because its search engine is so awesome, they did it by selling ads!  Adwords allows ordinary people to rent a link to themselves on search results pages for any search phrases they choose.  You only pay when somebody actually clicks the link!  This idea is what Google did differently, and changed the game.
• Third, recognize the problems with Google.  Sometimes it’s hard to find flaws with existing products.  Don’t think about what is wrong with the search process; instead think about what could be better!  It truly is the little things that make a big difference.  Think creatively!  I give a few examples from other web sites.  Amazon’s recommendations often think you want to buy a “blue” version of something because you bought a “red” version.  Maybe this works for them, but it does make me question their recommendations.  Ebay doesn’t allow you to only look at negative or neutral feedback.  You have to go through page after page of meaningless “great ebayer!” to actually find the truth.  These are problems you can only see if you compare what exists with how things could be.
• I’m sure there are thousands of computer scientists out there that could easily have implemented the search algorithms necessary to produce a competitor to Google.  When some idea finally comes along to compete with Google, it will also be easy to program.  Sometimes a project like this just needs to get a little push in the right direction, like Linux.  Linux didn’t get created to compete with Windows.  Someone just decided one day that they wanted a free Unix operating system.

Well, today I am DECIDING to compete with Google.

Post any ideas please.  I want this to be a collaborative thing with everyone out there.

A few short weeks ago I read the Yahoo compliance guide for law enforcement document as it was leaked on the internet.  As expected, not too many people in the general public have noticed or care about what is going on.  What does a company do when the public no longer trusts them?  I think Yahoo is on its last legs as a company.  There just isn’t much left for them to do.  It is only a matter of time before the idea of a homepage containing a ton of links stops working.  Advertising with banners just doesn’t really work.  It has been too abused by spammers to give any credibility.

So, what is left at Yahoo anyway?  The homepage where I used to check the top news stories still looks similar, but not as good.  Now the small news square at the center of the page covers itself with ads if you hover your mouse in the wrong place.

I would gladly have gone on using Yahoo for years if things worked well, but how can you trust a company that sells your personal and financial information to law enforcement?  Remember that the Yahoo compliance guide is written by Yahoo itself, not some crazy conspiracy nut.  Some person or group working at Yahoo wrote that document.  Yahoo PAID them to write it.  Yahoo probably paid lawyers to read it, and re-write it.  Yahoo pays people to answer the phones, and pays for the office space used to handle the law enforcement queries!

If the FBI, CIA, or NSA made a search engine, would you use it?

If your local police department ran an email and photo-sharing site would you use it?

Do big companies really notice if somebody has complaints posted on a blog?  What about if the blog is popular and the campaign of complaints is consistent, and uses SEO best practices?  Here is the situation:

1. I asked for only one gift this Christmas, a set of new headphones to replace my old ones that are no longer comfortable to wear.
2. My wife always worries that she will get me the wrong thing.  This year I had them on my wish list on Amazon.
3. My wife bought the headphones I wanted from a seller on Amazon.  You know, those people who sell new things on Amazon except that they aren’t really Amazon.  You find that this happens quite a bit with electronics since there are so many different items.
4. UPS delivered the headphones to the completely wrong address in a bad part of town, and now they can’t get them back.  Not that there is a chance that I would put crackhead-used headphones over my ears, for the record.
5. The delivery was made the Friday before Christmas, and it is now the 13th of January.
6. I got an I.O.U. for Christmas.

I read an article by Seth Godin today about complaining.  I want to be calm and collected about this situation for a change to see how things work out.  Even though it has been nearly a month, I am calling this day zero.  I will be keeping track of the situation on this blog both as an experiment in SEO, and in how well big companies get things done.

Day 0:

1. UPS officially has the delivery listed as a “negative” meaning that they lost the package.
2. Amazon doesn’t seem to care, since the headphones were not sold by them.
3. There has been no refund.
4. I have no headphones.

According to a few sources, or if you happen to leave your home for any reason, you might notice that gas prices are going up. Put away your beer bongs and your crack pipes, you aren’t imagining things!

For several days now, the prices have been going up, nearly to a $3 national average, and already past that here in California. Usually the rise and fall of prices is caused by supply and demand, but not now… not this year… It’s all about speculation! Fuel demand has not risen and supplies have not been cut. Speculation is all on the fact that it has been a cold winter and … well, that’s the only reasonable reason out there.

While it would be understandable out in the Midwest and East Coast, I am hardly freezing my ass in California’s 70-80 degree weather. What upsets me is how strongly gripped by they balls oil companies have over the US. People have grown accustomed to gas prices hovering around $3 and never again do I expect to see it near $2, let alone $1. I like driving; it can be relaxing at times when I’m not wrestling with traffic or retarded drivers. What I don’t like is how much the price to drive is allowed to fluctuate. What I also don’t like is that the prices can just go up on such a whim.

While the ‘simple’ solution is to research into alternative fuels and use more public transportation (public transportation in CA still sucks and there isn’t a huge push to develop it), special interests will find their way to cock block the government or researchers from pushing hard on developing it for the mass public. Same way insurance companies neutered the health care bill to the ground, same way the teacher unions killed off student vouchers, massive public transportation for California and alternative fuels will be lying in the graveyard until the complaints of the public become louder than the pockets of the oil companies. And seeing as how we’re giving them $3/gal every time we do anything and go anywhere, we’re going to need to blow out a lot of people’s eardrums for anything to be done.

What I think we can do to screw over Shell, Chevron, and all the other gas companies is to cut back on how much gasoline we use. If you feel like driving locally, see if you can’t walk, bike, or take public transportation to where you need to go. If it is a long distance, look into some public transportation or carpooling with others. The less gas you use, the less you have to pay for, and the less the gas companies get. Try using a motorcycle; $10/tank of gas is alot less than $40/tank – $100/tank for a car or SUV. If it’s cold, I don’t use the heater, I just leave my computer on. Electricity powered heaters are better and safer than gas powered heaters, just remember to shut them off.

To help the future, vote for more public transportation development… Southern California desperately needs it to alleviate it’s choked up freeways. The infrastructure isn’t there, but they are working on it. To fight the problems of today we have to be active; things aren’t going to fix themselves if we let the big companies have their way.

After traveling during the holidays I noticed that half of the airports I visited had free wifi as a holiday gift from Google (It probably would have been more, but some had their own free wifi already).  Am I supposed to believe that major corporations give holiday gifts just to be nice?  Of course not!  If the NSA or CIA decided to give everyone in the country free cell phone service as a holiday gift would you believe them too?

Never believe the REASON given for something being free!

The obvious conclusion is that Google found a way to make money from giving free wifi in airports.  Remember, with Google, You are the product! Google has never made it a secret that they store every bit of information they gather about you forever.  This is because they sell this information to advertisers in the form of adwords ads.  The advertisers who want customers of a certain type pay to get their ads shown to them.  The idea is that the ads are only shown to the most likely customers.

1. Google gives away free software and services as bait because they want you to give them information.
2. Google’s search algorithms are not optimized for relevance, but to increase their adwords revenue.  This just happens to correlate very well with their definitions of relevance, in case you are paying attention.
3. If you don’t use the Google search engine, they still keep track of your web habits through sites that use Google Analytics, which includes a significant portion of the web.  Don’t believe me?  Download the noscript plug in for Firefox and see how many sites use it.  Better yet, view the page source in any browser.
4. If Google is providing wifi, they know everything about your internet use.  Probably more than everything actually.  Obviously they know what airport you are at.  They know when you are there, and probably your general location in your terminal.  Add it all up and they probably can figure out what flight you are taking.  This means that Google has gone beyond using simple keywords to target ads to you!  Now instead of knowing that you searched for hotels near Chicago for example, they know when you will get there.  They know if you are taking an expensive airline.  They know if your flight is delayed, meaning you are probably pissed off and tired.

Seriously, be careful!  Google has never argued the fact that they collect data about you.  They say that your privacy is safe because if they screw up, you won’t trust them anymore.  Is there really a difference between information obtained from your computer by a virus, versus given away to Google?  It is the same information after all.  It’s not like Google is spending much money protecting your privacy.  They just want to make as much money as they can until the inevitable problems start to show up.  Sortof like Enron, right?  They know that eventually there will be a major security breach, they just don’t know when.  When I say major security breach, don’t underestimate what I mean.  A bank losing a few thousand credit card or social security numbers is nothing compared with what Google has on you.

• Expect Google to start providing wifi in more and more places.
• Expect adwords, and the use of keywords to disappear as the main method for generating revenue.
• Expect the new Google phones to be the most dangerous piece of technology ever released to the public, so far.
• Expect some sort of major money-laundering issues to come up.  Possibly in 2010.
• Expect to forget what Google “used to be like” and get used to the new Google that regularly sells information about you at a scale that would have shocked George Orwell.
• Expect that Google will patiently and calmly hold the world and its economies for ransom.
• Last one:  EXPECT THAT NOBODY WILL EVEN PAY ATTENTION!