Search Engine Optimization and Social Networks

Search engine optimization (SEO) has been a trending buzz word on the sites I've been visiting recently. I generally believe that there is a big market for business intelligence related internet work as there are so many people online today searching for so many different things. The key point is to make sure that your website is the most prominent to drive potential customers to your site and convert them into paying customers. In simple terms:

customers = site visitors * conversion rate

and what people in the internet business industry want to do is increase the number of visitors and conversion rate to make a business profitable. The problem itself is very fascinating as it requires a lot of data mining and profiling to determine where your customers are, drive them to your site and convert them into paying customers.

Consider that Amazon it self had $22 billion dollars in revenue for the year ending in 2009, that the whole total of online sales for 2009 is much bigger then that. Meaning that the business for improving online sales is big money.

For those of you have been on the internet since 2000, some of you might have noticed invisible text hidden away at the bottom of a site or in the html of others. The idea of optimizing one's site to rank higher in the index of internet search engines is an old idea. Google had an interesting take of the practice by both rating the number of external links leading into the site in addition to the content of the site it self when determining the ranking of a site thus improving the quality of sites they returned from their searches. I believe with the rise of social networks and social based information exchange sites, that the nature of search will change in the coming years.

I started thinking of first seed of this idea in a former post in June entitled Friends as Filters, where friends based on their interests will filter and pass along useful information within a circle of friends. With the rise of ease of sharing information through social networks, we find that social networks are having a significant impact on how information is passed along. We now have a new class of "viral" videos or webpages that become massively popular by dissemination through social networks instead of being ranked at the top of a search engine.

I find that Facebook is an interesting avenue for sharing information with friends, whether it be through the sharing of things they do or the interesting links they post. I do admit that I need to organize my contact list and start filtering people for interesting content for to improve the quality of information passed to me. But the basic concept is there, which is similar to social linking sites such as Reddit, (the now waining) Digg and more. Unfortunately as the number of users grow at these social sites, degradation in the quality of posting does occur, resulting in meme posts that are popularly voted up as a result to an appeal to the greatest unified demographic; in these cases, it need not be the majority controlling the "main page," assuming the demographics of a site is sufficiently fragmented enough to allow a minority group to have the most control over the popularity of the links posted. It would be akin to allowing a minority governmental party wielding majority like power inside a parliamentary house.

With this problem at hand, online communities have started reorganizing to support sub-communities where relevant information to sub-groups are passed along or posted only in that sub-group. What we are starting to find now is that things are a little different compared to the times of say Usnet when posted information was publicly available compared to the wall-gardened sites such as Facebook. What many spammers used to do was mine Usnet posts for e-mail address and spam everyone with advertisements. With the advent of more modern social networking sites, stronger linking of information is now possible however that information is not publicly available unless you are an app developer or the administrator of these sites themselves. What people are finding is the information in these sites is valuable when mined and summarized correctly. I think this will lead to a change from SEO related work to Social Network Optimization work in the future.

I think this field is still in it's infancy and to get information worth mining you are either going to get access to data in established social networks or create a killer application to create a network of your own, but I think there is good money to be made jumping into the social networking site for business intelligence applications.

Edit: Some typos and had 22 trillion instead of billion. I have a terrible habit with skimming material too quickly it seems.

Mastercard and Visa shouldn't be controlling who you can do business with

There exists file sharing sites that share that contain copyrighted content, but the idea that financial transaction institutions such as Mastercard and Visa working with the MPAA and the RIAA to preemptively block financial transactions with organizations they don't like sets a chilling standard with regards to the freedom of to whom you are allowed to do business with. We are seeing the same thing happen with Mastercard and Visa blocking financial transactions to Wikileaks because of the inconvenient nature of the material they are publishing online. Supporters from around the world are willing and wanting to donate to Julian Assange and his cause through both financial and technical support.

There is nothing more democratic when it comes to people voting with their money, which is exactly something that the US claims to be a staunch supporter and defender of. Unfortunately this simply isn't a battle of ideals, free markets or rights. It's simply organizations trying to change the game to keep other people out of their political/economic turf.

Facebook and more intrusive ads

I generally use Facebook to keep track and stay in touch with friends. The biggest motivations to get into Facebook was to get reconnected with people that I had lost touch with and be able to keep up with them. I believe that the social graphs of people in today's society is simply much larger compared to what it was before as a result of the internet.

I've recently upgraded to their new profile and was aghast with the prominent display of ads on my profile page. To me, it's a terrible eye sore and I am not very happy about having a personal space being used as an advertisement space. That is a digital line that I would much not like to have crossed. I have since installed and enabled the adblock add into firefox and the ads have since disappeared. For now, using ad-block is a viable solution, but there ought to be better ways of maintaining a social graph, content and access instead of using a closed platform. One of those alternatives is Disapora, which is still under development. It will be interesting to see what happens from there.

On Digital Money Usage in Japan

Of of the interesting thing about about Japan is the use of digital money. I used to think that digital money was close to inconceivable because I much prefer the feel and sight of having cash in hand when making payments. The physical and visual queue of using money is that has greatly helped me in keeping my spending in check. After moving to Tokyo, I've become accustomed to the prevalence of using digital money.

The cafeteria at my office operates on a digital money system. We have several thousand employees at the research center so the lunch lines can get pretty long if you get to the cafeteria right at noon. When I was a university student, it would take a while to get your food and then take another long while as everyone fumbles with coins and bills to pay for the food. Credit cards are one solution to that problem, but I believe that digital cash cards are superior compared to credit cards.

One electronic cash card system being used here is called Edy, which is a wireless rechargeable cash card that are also integrated into some cell phones in Japan. These cash cards can be charged using recharge kiosks where you can place your card or cell phone on to and deposit money into the card. Another interesting feature for cell phones is that you can log into your bank account from the cell phone and have money wired into the chip in phone. One point of risk for the system however is hacking of the cards to digitally manipulate the amount of money stored. I am not entirely sure about the security features used in the card to prevent people from tampering with the device, however.

The Edy cards are currently accepted at nearly all convenience stores and some retail chains, so I would not say that market penetration of the cards for consumer usage is not that deep. However for mass transit usage (trains, buses and etc.) pretty everyone uses these cards as you do not need to look up train fairs and pay for it every time you go through the gates. Throughput of people in Japan through the train gates in Tokyo is quite amazing as a result of this.

The one nice thing about these digital money cards, is that they are not necessarily tied to the identity of the user, for example in the case of a credit card. The loss of a credit card usually incurs the headache of contacting the credit card company, deactivating the card and waiting for a new card to be made and sent to you. Without having this identity connection, a card loss is simply equivalent of physical loss of the money, meaning that you could just buy a new card (which is about $10 ish), reload it and start using it immediately.

One of those things I've been getting annoyed about recently is the size of my wallet with the number of cards I carry. Women, in particular, I notice have huge wallets to carry all the point and membership cards they use for shopping. As of recent, I've been thinking about how nice it would be to be able to carry a much less boated wallet with me.

The financial incentives for managing a digital money system is obviously there as money charged into these cards can be placed in a holdings account where the institution can make 1~2% interest in very low risk investments. When it comes to creating digital consumer ID cards, I am not entirely sure that there is sufficient incentive yet to make that jump yet, though I believe that work is trivial.

I would expect that ID information will eventually all become digital and embedded within an ubiquitous device like a cell phone. The interesting thing to ask is what are the risks of digitizing all ID and monetary data? Furthermore, I would wonder what kind of system would manage it? For it to be socially acceptable, I would be willing to venture a guess that the management system would have to be a decentralized one. Maybe we will see that transition in the next 5~10 years after the first "killer application" that drives this transition is created.

Intangibles

I am oft known as a frugal person, in that I don't generally like spending money if I have to. Optimizing/reducing cost is one of those things that I've been good at doing; however there are things that do go missing from the equations, the intangibles.

Tell me that you need a hammer and I'll tell you where you can get a sufficiently cheap and good one. Tell me that you need an affordable computer and I'll set you up with the specs for a system and where to get the parts for the right price.

It has been mentioned to me recently (and repetitively) that apparently, I need to "improve my fashion." Like in this XKCD comic, "my normal approach is useless here." I was dragged out this weekend to shopping in search of new threads. Why is it that one piece of paper scribbled by Vincent Van Gogh is far more valuable than something scribbled upon by a random person? Art, unfortunately is one of those strange realms that numerical analysis is rather useless upon.

The one thing I know is that I spent a lot for shoes, pants, shirts and a jacket. The other interesting question is, what interesting intangible benefits of "looking sharply presentable" have? Apparently, I have feeling that the answer would simply be best described by as "a lot."

Friendship map of the world

I've always been a fan of finding interesting ways of mashing up data from the internet. One project I did was downloading performance ratings of CPUs and combining them with pricing data to calculate price/performance ratios before selecting a part.

Recently, an intern at facebook stripped out friendship data from their servers and made a map of connections between cities, by mapping what city they were from into geological coordinates and drawing lines between the points. After some artistic manipulations of the lines, his work resulted in this.


A very cool map indeed. Some interesting notable dark spots are South America, Russia, Mongolia and China, parts of Japan and along the Canadian and US border. If you're interested in the details of this map, a link to the original facebook article is available here.

What would you pay for a set of games?

The band Radiohead once did this and this is the second time that an indie game developer is doing this. It's the humble indie game bundle, where you get to choose how much you would pay for a set of 5 games and they are posting live information on the how much people are paying and by what platform. The trend seems to be the average "donation" is Linux > Mac > Windows. Where Linux users pay 2.5x more than what people pay on the Windows platform. Ironic that people that use a free OS would pay more for software.

An interesting article on if you should get a PHD

The link to the article is here. It's a good read and I haven't finished reading all of it yet (not that it's long).

Web based programming is the next big thing

Programs used to work in isolation on a single PC somewhere before the internet became prolific. Computer science classes at university used to be all about programming in C or C++ or something like scheme. But now with the advent of the internet, most of the programming skills we learn at universities are obsolete.

The thing I remember the most about my time in programming classes was dealing with how to store data, how to process it and then how to display it to the user. I still think that these things are fundamental to the core of programming, but I think there is a big demand gap when it comes to finding skilled people that are able to get data to feed their programs, process large data sets and come out with meaningful results. This is the stuff that business intelligence is made out of.

Over the last few years, there has been a big transformation in the world of programming, databases and languages have now been developed with the paradigm that given certain data, there are specific functions calculations that you want to run on all the data and summarize them into a single result. This thinking framework is otherwise known as MapReduce. The great benefit of this methodology is that is is also functional programming friendly, meaning that you can define compound functions that you want to run on a specific data set instead of dealing with loops, counters and how you are going to save the data. The basic idea is that you make a list of data, process it and out comes a list of data. Then after that, you reduce that list into a single value or something.

Modern languages languages like Python, Erlang, Lisp and more are designed to handle programming like this (in additional to dealing with the usual procedural type programming and object oriented programming). Older languages like C or C++ can do the same thing, but you have to build the functions and the scaffolding to do this on your own or call libraries to do this kind of programming which make C based programs longer and more error prone to write.

One of the reasons why I was surprised in an earlier post to find some large corporations doing data processing still wanting C++ programmers. The other added benefit of using more recent languages is that they also have libraries to access web based data built in, making it easier to feed applications with more data to process. The of the most famous startup companies are the direct result of this effect. You won't learn most of this stuff from your ordinary university class, the only way to get into this stuff is probably on your own.

Learning how to work with open software projects

I've seen some interview questions and job postings that mentioned that working on open software projects is a nice to have skill over the last few years. I never understood how much of an impact that working on an open software project might have on a person's ability to code until just recently.

I have been following the programming scene in sort of a cursory manner, mainly through reading at hacker news, which is a good site for programming and web development kind of information. What has been coming up repeatedly is the development of a program project hosting site called github, where the collaborative projects are hosted. The ability to code well parallels with learning to write well; in that reading good code helps one write good code.

I have cursory looked at sites like source forge in the past to download code to look at, but one of my biggest problems is trying to figure out how large projects are structured and what code is where, doing what. I used to do most of my coding through a simple text editor before because configuring and trying to figure out these modern fanged integrated development environments (IDEs) would get on my nerves when I wanted to learn and compile something simple. I've used to code through a simple text editor for the last 2.5 years until realizing that the debugging facilities and managing large chunks of code became time consuming and brutal.

I have since switched over to the freely available Eclipse IDE with an integrated debugger and the ability to read locally created libraries and allows for auto code completion. It is a godsend compared to what I was doing before. Commercial packages like Microsoft's Visual Studio probably has functionality like these built in, but learning coding and figuring out IDEs on your own at the same time is a little much. The great thing about Eclipse and some of the new open source IDEs is that they are starting to add functionality to connect and download projects from code repositories from places like github. I have just finished configuring IDE to do just that and downloaded the libraries to a data-mining spider called scrapy. There are other libraries available that does the same thing but I have the intentions of learning how the program was coded, the programming styles used and possibly using the project itself for my own data-mining projects. After looking at some of the code, I can understand how some people would look highly upon people that participate with open source projects as they are exposed to working in an environment with many people, many lines of code and looking at a lot of good and bad code. I will be spending time at looking at more programming projects to further hone my own skills. The idea that companies look at people with open software experience brings up the interesting concept of profiling people and their skills through secondary activities and traits. It's an old idea however, but it would be interesting to see the correlation of coding skills to open source project participation or say a person's affinity to knitting and a (negative correlation) to say, sky diving. Unfortunately, statistics when applied like this is like opening a can of worms.

Can't make heads or tails of silver

Apparently there seems to be a campaign on the way to crash J.P. Morgan via the purchase of silver as they supposedly hold a short position of 3.3 billion ounces of silver and are currently under investigation of silver price manipulation. In the last few months, silver has been trending upwards for the last 3 months with a 47% gain. Gold hasn't seen that much of a price increase.

Silver ETF price for the last 3 month

When it comes to financial information on the internet, there is a lot of garbage out there, especially for a hot topic like gold and silver. The leading argument to buy silver comes from a guy by the name of Max Keiser, in general, I don't trust financial pundits and whatever they start talking about, I generally tend to ignore but make a note to look into on my own.

The burning question that needs to be answered is, "is silver relatively cheap or expensive right now?" I've added to snap shots for the last 10 years on gold and silver prices from kitko.com.

Price of Gold

Price of Silver

Metal prices have gone up significantly over the last 10 years, but I'll skip the commentary on that:

Gold in 2000: $300 (approx), Silver in 2000: $5 (approx) Ratio: 60:1
Gold in 2010: $1414, Silver in 2010: 29.58 (Dec 6 prices) Ratio: 47.8 :1

Historically, it seems that the price of silver has been much lower compared to gold in recent times and I find it hard to believe that this ratio should suddenly change. I've also been looking at the relative production rates and total produced numbers of silver and gold and get wildly different ratios compared to the prices (of course the numbers themselves might not be reliable too).

From gold-eagle.com: I calculate that by 323,476 Tons (by 1999) have been extracted. I guestimate that about 450,000 tons have been mined by 2010.
From wikipedia: They mention that 165,000 Tons have been mined by 2009.

Meaning that there is a 1 : 2.77 availability ratio of gold to silver. I've also looked at the production ratios and the numbers work out to be 1 : 4.8 for gold and silver. I think that stockpile availability might be the key factor and the difference between supply and demand might be causing the price differentials between gold and silver, but I still can't figure out the reason why gold is that much more expensive than silver, presently and historically, unless the numbers I've been looking at are bad (which is a possibility). Personally, for the bigger price differential between gold and silver to be supported, there ought to be a greater differential in availability between the 2 metals.

The other forward looking argument to be made is, suppose that people are buying silver to crash J.P. Morgan and that it does happen. And then what? I would say that eventually everyone would have to sell their stocks of silver to get their money back out of it thus dropping down the price of silver. The question is, when do you start selling? If you can't figure that out then you shouldn't be buying in the first place.

It'll be interesting to see what happens, but I have a feeling that there is a bubble in silver happening right now.

Addendum:

I'll add supply data from kitco.com here since I think the data from here seems a little more reliable and up to date.

Gold supply (2009): 117 Million Ounces, Silver Supply: 810 million Ounces Ratio: 6.9
The ratio is still fairly small compared to price ratio between gold and silver. What is the pricing mechanism for gold and silver? I still don't know.

Reducing distractions and the 10,000 hour rule

I am in the middle of a transition period in life; I've got another 6 months till I turn 30 and the big question of "where do I go from here" that I need to figure out. One of my colleagues turned 40 a few months ago and the only thing he was muttering the whole time was "I just can't believe I'm an old man." The interesting thing about my life is that I get along well with people that are older than I am; it's always been that way when I was young and the best part of having friends that are older than you is that you get a glimpse of the future so I can prepare for it now.

As of recent, I've been thinking much about my future; or at least, a strategy for one. Getting prepared for the future isn't like getting prepared for a trip or a disaster, where you just spend a few days going out and buying what you need. Preparing with skills, knowledge and experience is a different game, where you pay for it in the currency of time.

The sub-prime crisis and the following recession/depression fiasco has wreaked indiscriminate financial catastrophe to all walks of life; the young, the middle aged and the old. What was interesting was how each group was affected by the economic downturn. For example, the young cannot find jobs, the middle aged are have debts to deal with, and the old had their savings hit. What this economic downturn has taught me is that you need to keep a good backup plan to protect yourself from being stuck between a rock and a hard place, especially if you've become old (or transitioning to being old) and had your savings wiped out (naturally, you would have been investing in "secure" investments).

So where does this all fit in with me? At this point in life, I've come to realize that with growing age, the number of options I will have available will start to shrink in then next decade-- the number of new fields I will be able to jump into and get really good at will dramatically shrink. So it is now that I should start looking into explicit skills that I can enjoy and be really good at to make a good living. It has been oft quoted from by Malcom Gladwell's book "Outliers," that the number of hours it takes for a person to become an expert at something is 10,000 hours.

Doing the math, to become a master it will take:

• 27.4 years at 1 hour/day of practice
• 13.7 years at 2 hours/day of practice
• or 6.8 years at 4 hours/day of practice

This calculation also puts a lower bound on how long it takes to be really good at something and also the number of skilled expertise that a person can have over the course of a life time. The other thing is the earlier you have more skills, the more they will pay off in the long run. Meaning, if you start training hard by the time you are 50 years old, you might finally become an expert by the time you are 60, however that doesn't leave a lot of time to put those skills to use.

The other interesting thing about skill sets is that they also come outdated as time goes on. Someone that might have been a really good switchboard operator/manager in the early 50's would find their skills useless in today's modern world. Or perhaps a little differently, some computer gamers might find their skills in Starcraft 1 be outdated by the release of Starcraft 2. You might have some transferable skills, but you won't be an expert off the bat. This brings up the interesting point that transferable skill erosion occurs with every new generation of technology, the further along we go in generation, the number of skills we can transfer will negatively compound. Though it might be a little early for me to start thinking about this, it is something that I should keep on mind.

So here is the thing, suppose that I do practice something at 4 hours/day, it will still probably take me 7 years to be really good at it. What it also means is that if you want to be good at something in a reasonably good time, you really don't have a lot of time to waste.

After coming to this realization. I have decided that I am going to institute a system to reduce the number of unproductive distractions so I can spend more time learning. Mainly less checking of news sites, less youtube, less computer games and etc. This doesn't mean that I will be aiming to live a boring life, far from it actually. What it means is that I should more effectively use my time by doing things that brings me memorable value; which could include picking up skills or spending time traveling to somewhere exotic, learning more about wine or whatnot.

I think there was an Indian saying that went like this:

When you are young, you make your habits.
When you are old, your habits make you.

Time to make some productive habits while I still can.

People still use C++!?

I find it amusing that people still use C++. I am a big python programmer, which was a transition for me from C++. The python programming language is just far more extensible, easy to program and dynamic variable typing is so easy to deal with that I just can't fathom that many people still program in C++. Many of the best startups on the internet don't touch C++ at all. Do I brush up on old skills or keep moving with the new stuff, a bit of a dilemma.

The problem with disclosing income

I've been doing some reading recently about the issue about disclosing one's income during an interview. The problem is that it give the employer an incentive to underpay you knowing your payment threshold. I've been on both sides of fence recently, knowing what pay looks like in the finance world thanks to a few contacts I was able to target for price at par with other people in the industry. Unfortunately things didn't work out as they had someone without a job that was willing to take the job, but that was OK. I would much rather not take a job if I knew I was going to be underpaid -- because one of the reasons for moving to a new position is to leverage your skills for a higher salary.

Since that deal had just recently fell through, I've just been introduced to some recruitment agencies and in the process of sending out applications for new jobs. I decided this time, that I would see what happens if I put something close to my real salary. I was soon contacted by a representative that would pass my resume to a few other positions. The guy quickly asked about my salary inspirations and I gave him my ball park number, he immediately balked at the idea of me looking for a 40%~50% salary increase, citing that I had a much lower salary listed. The moral of the story here is to not disclose your salary or put down a higher number.

I had qualms at first about inflating my salary before, but after seeing this reaction, it is an unfortunate necessity of the negotiation process.