Why you shouldn't try to become the perfect trader

I've been going through historical data in the Dow Jones recently to try and better understand past trends. I've mentioned before that I generally try to stay away from technical analysis but I've decided to put away that hat and decided to see where this takes me. The one good thing that looking at historical data is understanding the scale for change, things are measured in months and years instead of days and hours. I have a terrible habit of looking at daily market data, thinking that motions in the day are indicative of where things are going to be in the long term. So I'm going to stop looking at the market like that (though it can be fun sometimes... sort of my equivalent of "watching sports").

I've simply started plotting through the Dow Jones looking at annual returns year by year. Looking at annual returns is a really arbitrary thing to do since I am picking a price at some "random" date and comparing the ratio of it to some other date in the future. I am sure that picking different windows will have different values but what else am I to do, eh? Not a whole lot until I figure something out, but for now this is it:

I've grabbed annual data back till 1971 and the funny thing about looking at 40 years or annual data is that there is so little data to look at. The first thing that pops out at me is the all negative change years occur after a +10% change year (though not after all 10% years). The second thing is that 5 out of 7 negative years turn positive the year after. So what would you do in this case? One could surmise that it might be a good idea to take out some money after a +10% year and put it elsewhere as a contingent against a negative year (Ie. Bonds or something). The next thing is that you might once a negative year occurs, you have a good probability of catching a positive year after a negative year. Something like that could work but the problem is that you will still need to optimize against having 2 consecutive bad years. There is also a period in the 1990 where there was nearly no negative growth years, but let's say that you've optimized your strategy to factor that in too.

Now suppose that based this 40 years of data you've made a fairly optimized portfolio where you take a small chunk of money during good years and put them into bonds when you make more than 10% when the new year hits and made a rule that if there is a negative year, that you would plunk (say) about 60% of the money you had in bonds into the market during a bad year to catch the probability of a good year the year after and it left you with enough funds to deal with 2 consecutive bad years you could leverage yourself since the probability of having 3 bad years according to this data is zero (the alarm bells should already be going off here).

Suppose that this strategy worked well against this data, you might be fairly tapped out if there are 2 consecutive bad years and the strategy provides for very good gains. Unfortunately, if you look far back enough in history, there is a period right after 1929 where there was 4 consecutive bad years. If a black-swan event like that happened, everything in your portfolio would be gone. Unlike poker, you don't generally have "rebuys" in the stock market (unless you have really rich and possibly foolish relatives), If you're going to want to play this for the long game, developing a methodology to understand what is going on around you, knowing your options, knowing your limits and knowing what exactly you are willing to risk is important. Having good well founded rules to maximize your success are important and you should, in general, play by self set rules that are designed to maximize your chance of success (and though, just like poker, success isn't guaranteed). And just as a note, there are 4 years of consecutive negative growth around 1929 that might have wiped this strategy out (then again, it probably wiped out nearly everyone during that time). In addition, you may want to have a contingency to catastrophic events.

I had this knowledge internalized in me in poker so well that I had forgotten to be aware of my general rules that made me a generally strong player. Most of my poker playing rules were also mathematically generated too, the game of poker (un)fortunately is really easy to analyze. All you need is some general understanding of probability, expected value calculations and knowledge of permutations and you pretty much had it made. The stock market is different and choices and information is so much more vast such that it nearly impossible to understand the entire system... though I still do try (though I am off on a tangent).

Rule number 1, though, it to have a set of rules to play by and more importantly, have a set of mental interlocks to prevent yourself from going "tilt." Getting into the stock market will make one incredibly emotional. A bad stock market period is like PMS for men (that is not to say that only men participate in the market) and you honestly need to get a hold of yourself during those time because you are going to be prone to doing stupid things.

This coming of the second crash (?) has given me a great opportunity to better look in to and handle crashes. The first stock market crash that I have ever had to have deal was the 2008 one, and my, that was a doosie. I wished I was old enough to have participated in the 2000 dot com bust, because I would have been much younger and had a lot less money to lose. Unfortunately, I was too young and had very little real capital to my name during that time (and the cost of trading was way too expensive during that time; $29 for a single trade!). Anyways if/when I do have kids, I am giving them money to blow in the stock market, they might as well learn to lose everything early and fast (and I hope to not make chronic gambling kids while at it too).

Fail Safe (avoiding gambler's ruin)

Unlike other games, the stock market is the kind of game where you play on one "life." The idea that you're supposed to be in the market till about 65 ish (or longer) and not screw up and lose nearly one thing is actually daunting. It's like pretty much saying that for 30-40 years that you are not allowed to fuck up. But that is pretty much it, lose once in the middle of it and you are possibly screwed.

I've come to the realization that optimizing for gain and building a robust portfolio are (to me at least) opposing factors. Optimization is basically trying to allocate resources perfectly to come out with the most perfect result as possible (which is easy in games, hard in real life) because it can equate to doing things with minimal margin for error and going for possibly unrealistic perfect results. When it comes to investing, you are never going to get the most of that up swing, nor are you going to know when the market has hit bottom. The question is not a matter of making the perfect play (because it is impossible), can you make decisions such that are "good enough"? The next important thing is that can you make consistently make "good enough" decisions? Because if you can, then you are on your way to becoming a good investor and I think this is often an overlooked point.

The important thing about investing is being able to quantify the risks of every decision that you make and have a general idea of what you are going to do if things turn bad. How much loss are you willing to take? if losses do occur, are there other assets that can balance against that loss? Is the portfolio setup in a way that you won't be psychologically affected if things go to bad (or good?). Risk quantification in a portfolio is really tough and it is a really fuzzy science but it is very necessary. One of those lessons learned from my crazy crash and burn in 2008.

How am I faring?

I think I was lucky this time because I wasn't overly invested in equities and though the equities that I did hold lost a lot of value, I am sleeping better compared to last time. The other thing I should note was that I was a little impatient in getting back into equities a few months ago when it seemed to me that things were going to start taking off. The equities that I did end up buying were over priced because I thought things were going to be a little better than they currently are right now. I should have been more patient but that is the way how hindsight works.

I've also thankfully kept a good amount of cash out of the markets and will be patient for things to cool down in the market before looking at getting back in. The volatility will be here for a few months until Europe sorts it self out and people get over the S&P downgrade of the US (which ironically, did something good for US treasury sales). There will be plenty of sucker rallies and swings coming up... but then again, that is already pretty normal for the stock market.

If you can make sense of the following data, then you deserve to be a millionaire.

A look at some past crashes

I had some spare time over the last few days so I've been going through some of the Dow Jones historical data with some programs to study volatility during some previous stock market crashes. I first plotted out daily percent changes in price and volume to look for large swings in price and volume. The result of that plot is provided below:
Looking at the percent change in stock price, it become fairly easy to pick out periods of time of high volatility and we see a few points in 1987, 1989,1997,1999-2003,2008-present. By the looks of the graphs, it takes a few months until volatility subsides. Though as of today, it seems that the stock markets are up, I highly doubt that the choppiness seen in the Dow last week will be the last.

The % change in volume plot doesn't capture well changes in volume since it could simply mean that volume could have suddenly risen and remained constantly high during a crashing period. Though from the plot, we do see that volume can vary quite wildly.

Given the % change in Price plot for the Dow Jones, I had a program go through the data looking for points in time where a > 5% decrease in the index occurred and extracted 4 months of data before the crash and 6 months after the crash to get about 10 months of data. I also had the program mark points where a > 5% decrease occurred and came out with 7 plots:

Black Monday Crash

Friday the 13th Mini-Crash
Asian Financial Crisis

Russian Financial Crisis

Dot-Com Bubble

Sub-prime loan Crash

US Debt Down grade Crash

The data seems to indicate that during the pre-2000 eras of crashes, volatility was far lower compared to what we are seeing in the markets now. This is probably the result of the HFT algorithms going at it in the market place. The most recent crash is showing far more volatility compared to all but the 2008 stock market crash as we have seen in a single week, some very large price movements in the Dow Jones Index. There was also a large spike in trading about a week before volatility picked up and I am not sure what this represents.

Not really sure where the volatility is headed from here, but there could be more on the way if things this time around play out in a similar way to 2008. We'll have to wait and see I guess.

Where does "net worth" go during a stock market crash?

Understanding your "opponent's" trading account is akin to keeping track of the number of chips your opponents have during a game of poker-- the amount of cash your opponents have will have an impact on the way they play the game.

One thing about stock market crashes that fascinated me is trying to understand where the money and capitalization value goes. I have been working over the idea of the existence of "phantom capitalization" in the stock market as a result of stock prices being bid up and I was interested in trying to understand the mechanics of a stock market bubble. So I was basically trying to model a feedback loop were people start pouring more and more money (and leverage) into the stock market, thus creating the illusion that everyone has more equity via "phantom equity." The process repeats until it is unsustainable and things fall apart.

I believe that there is a disconnect with this mode of thought since people are fooled into thinking that this equity is real and the harsh reality sinks in when the inflated stock prices tumble which wipes people out. The real objective of participating in the stock market is to buy and sell stocks to eventually cash out with more money than you started. By playing by any other rules instead, I would argue that the investor is fooling themselves when participating in the stock market. Unfortunately, the calculation methods used to value equity is somewhat faulty and is prone to enticing people into inflating stock market bubbles.

The first thought that came to mind was to try and simulate players and their interactions in the stock market; trying to understand the impact of their buying and selling of equity and how it would look like in their trading accounts. Obviously, this is a pretty hard problem as the real stock market has many individuals and tradable securities. Going for the full monty is obviously too complicated so I decided to sandbox the whole issue and have 2 players with cash and a single tradable stock which both players owned at the beginning. When one player issued a buy order, the other had to sell at the same price and volume and by using this simple model, I wanted to see what a stock bubble would look like under this isolated situation.

I managed concoct a trading pattern between 2 players where they could buy and sell and equity at increasing prices while keeping the amount of money and the number of stocks they had on hand constant. However, through the act of trading and using the standard method of calculating the "average action price" of each person, I was able to inflate the price of the stock and thus was able to create phantom equity for each individual. I have included a screen shot of the simulation I performed in the image below (click for the full sized image).

In this rather simple simulation where I kept the volume constant, I noticed that the swings in the "cash" column got larger and larger as the price the stock increased. In more realistic situations, I would presume that the volume would taper downwards to reduce such large swings in one's capital account. I would surmise that this also wouldn't apply to just individual investor but is also true for computer systems engaged in high frequency trading. But the main point is that it is possible to observe the inflation of networth of 2 parties trading in a single market and that it should be possible to extend this model to a many people market with many different stocks.

For the above calculation, I didn't take it all the way to the point where both persons were leveraged to continue bidding up the equity price, but it should be quite evident that the swings would get larger until the point where neither person A or B would be able to leverage themselves any further. Supposing that a dormant person C existed, any sell off by this person would result in a catastrophic crash as margin calls are made.

From a technical trading standpoint, I think it would be an interesting exercise to find ways of watching the volume and price as an indicator of bubbles in the stock market. One concept would be to watch for decreasing volume while prices rise as an indicator of a bubble since higher prices will equate to lower volumes as a stock is traded up. I would also expect that the converse be true in that volume would likely increase when stock prices are low since it takes more of the same stock to move the same amount of money.

What I am very interested in trying to figure out as a result of this simple calculation is an empirical method of guessing at the "action prices" of other participants in the stock market. I have a hunch that people will have a tendency to sell off their stocks after turning a profit and will likely end up spending their profits on buying other (inflated) stocks. The whole process continues until a bubble occurs and it pops, leading to the destruction of phantom equity.

One method of making money in the stock market is simply knowing when to get out and staying out when a bubble occurs. And the trick to understanding when a bubble is occurring is figuring out when there is too much phantom equity vs the amount hard cash available.

The Unwinding yen carry trade?

The yen is skyrocketing recently against the USD, CAD and the AUD over the last few days. The chart for the JPY vs the AUD is especially astounding as people are unwinding their carry trades. The recent spike is astounding. The other thing, however is that I am quite certain that the rising valuation of the yen is unsustainable against other currencies and there should be a correction happening eventually. The question is at what price and how volitile things will be during this time period.

One thing that I appreciate after coming back from Cambodia and Vietnam is an understanding of the bartering system, where the prices of goods is often bartered between the customer and the store clerk. Prices do vary between shops and it is very interesting to see how prices can be very fluid between places. Everything is driven by the sentiment of the store owner, the customers and other market circumstances and this experience is equally applicable to the stockmarket.

One intersting observation that I made about the merchants that I bartered is that they liked to ask where I was from. Basically, what they were trying to discern is what price would I be willing to buy something by the living standard of the country that I lived in. A Chinese person might price things differently from an Europeian and etc. When it comes to the stock market, instead filtering the buying and selling patterns of people by country, one would need to understand the trading motivations of different financial institutions. Meaning that different people would have different ideas of what a fair price for the same financial instrument.

Anyways, I am very tempted to sell more of what yen I have into other currencies, but the question I am trying to figure out is how long will the sell off in the stock market will continue. I believe that the yen will continue to strengthen the worse things get in the stock market. The question is when will the sell off stop, at what price will people start selling yen and etc. All of this is really hard to figure out.

When it comes to the financial market, one cannot look at companies in isolation but also other traders and their motivations in the market. I have a new appreciation for that and very interested in developing tools in trying to understand the financial conditions of the other players in the market. As in poker, you have to be able to keep track of the number of chips everyone else has because how people play is also dependent on the number of chips they have.

In Cambodia

This is probably out of the blue, but I am in Cambodia right now. My work had a scheduled 1 week shut down of the office and facilities as a result of power conservation efforts in Tokyo. Apparently, companies have a sort of power usage limit between now and until mid-September and going over the power limit will result in fines based on the number of hours a company is over the limit. There are other details but I won't bother getting into them as it will probably detract from the main contents of this post, as in me being in Cambodia.

Dispite backpacking through South-East Asia 3 years ago, Cambodia is one country that was on my list to goto, but I never made it to because of time constraints. This time around, with the 1 week holiday from work I would go and hit Hanoi and Siem Reap to see the Angkor Wat temples. After a bit of a turbulent jounrey from Vietnam to Cambodia (not to mention that they also block facebook there), here I am in Cambodia and I have some pretty positive things to say about the country.

The people are incredibly warm and after getting out of the airport there weren't a bunch of taxis waiting outside waiting to over charge travelers to get into the city. We also had a tuk-tuk (a motorbike connected to a sort of trolley) that waited patiently even as my flight into the country was delayed by 2 hours. Compared to other SE-asian countries, there is a lot less price gouging here; then again, the cost of living here compared to other countries is so much cheaper that even if they raised prices, I might not even notice.

The price for a dish or something from a local stall ranges in the $1-$2 range, a foot massage for 30 mins is $2.5 and a full body massage for 30 mins is $3, just to put things in to perspective. Staying at really nice hotel runs for about $30/day, which is really reasonable. My imagination runs wild with the idea of running off here and not returning to work for simply years while staying out here.

The other nice thing about people here is that they are generally good natured and even if they can't get a sale out of you, they still are nice to you. Which is a rather nice touch. One of the guides I met is only but a teenager but has managed to learn a really good level of Japanese through 1.5 year of study and acts as a guide for many Japanese tourists that come out here. The determination I've seen in some of the youths here is simply astounding.

There is one sad thing that I've felt while being here is that there is a huge influx of tourists in Siem Reap and it feels like that many of them are just here for the cheap thrills of a country with a really low cost of living- there are some people out here that I've see that I would have a hard time imagining getting anything of value from visiting the angkor wat temples and at the same time, it feels a little sad that the locals are doing their best to cater to these people through the creation of bars and restaurants to turn a profit-- a perfectly good environment being ruined by the influx of money through the western world... and that was probably the most profound insight I had while walking around the city today. I took a bunch of photos, but haven't had the chance to download and sort them yet. Hopefully, I'll have some time to do so later on.

Being shipped off to Kyushu for a few months

The current phase of the project that I am working on has ended and as a result of that, it seems that my expertise are required elsewhere in the company. Crunch time is coming up for a special Sept deadline and I was just called in by the boss, yesterday to learn that I am being shipped off to Kagoshima, Kyushyu for 2 months to help troubleshoot some process related issues.

They'll be putting me up in a hotel for the first week and then a mid-term housing arrangement while I am there. I'll likely be working a few graveyard shifts as well... the compensation for taking this job is still being hammered out but here is hoping that I get some good spending money out of this.

Working on a few other things on the side as of the moment and the weekends have been really busy with events.

In other news, I'll be taking a 1 week vacation in the first week of Aug to visit Vietnam and Cambodia. Posting will remain light...

We are in for an interesting FX ride

Been following the news a little bit lately and it seems that Greece and Italy are having debt issues and the currency markets are going nuts.

The US will need to pass a bill to raise their debt ceiling on Aug 2 and what exactly will happen there is uncertain but I am quite sure that they will raise the debt ceiling, because it will have significant impacts on their ability to raise money in the future if they go into a mini-default.

As a result, I've just been looking at the currency charts recently and the yen is jumping upwards in the midst of this uncertainty. I've already diversified some of my funds out of yen an may look to purchase more foreign currency as I think that the markets will be fairly unstable.

It will be interesting to see how things will play out over the next few weeks, but I'll be lining up a cascading orders in CAD and AUD should the yen spikes up some more.

What does it take to run an E-business

I've been mulling the idea of having some parts fabricated and sold online recently and got down to considering the challenges of logistics and putting together a small site to see what happens. It's more of an experiment than anything else but having sold some X-box's though E-bay before, I just thought: "how hard would it be to create an online store and be able to accept credit card payments for something."

Been looking around here and there but a cursory search indicates that one can open a merchant account for about $25/month through paypal and deal with them skimming off some of sales. Modeling selling things online is also quite fascinating and what I noticed is that sensitivity to profits is very high when margins are low. The most important thing is learning how to create a landing page and figure out how to drive traffic to a site to gague interest before starting.

Premature Optimization is Bad

The title almost sounds dirty, but still, one shouldn't get too excited and ahead of them self when it comes to implementing programming solutions. One of the biggest challenges to programming is understanding the full scope of the problem that is dealing with. When I mean "problem," I don't mean it in a sense that something is broken, but in the sense of understanding the "specification of the system" of what one intends to build.

I am currently putting together a more advanced parser that can read text and extract values out of it from html. With the advent of content management systems (CMS), most of the data displayed on the internet follows a specific layout depending on the page. Writing parsers can be a tedious process as you need to do several things:

1.    Effectively isolate the block of text want to analyze
2.    Write a parser to target and extract the information you want to get, and
3.    Create a data structure to save that data

This process is time consuming because I would have to write additional code to parse different values and change the data structure to save everything. For every additional dependency there is in code, the probably for error goes up exponentially (a gut feeling). The less dependencies between modules there are, the less errors in code that you'll probably get. The best kind of code is code that automatically adapts itself to whatever you're doing... but I am going off on a tangent. Right... dealing with premature code optimization.

I have come up with an idea using html templates instead of writing code to parse data out of an html block. Writing procedural code requires one to think of a strategy to get at the data wanted and then codify the process in a program. That means that for every block of html, one would need to write code to get at the data. I've done that before and it can be a time consuming process (which can be made somewhat easier with the use of parsing libraries however).

Using templates it becomes pretty easy to specify the structure of the text and target certain sections of the data using keyword markers. Editing text and converting them into templates can be somewhat tedious but still far easier than writing code to extract data. The work required is just turning parts of the template into wild cards and entering keywords to certain parts to get at the data and I really like this solution, so much that I decided to write code that would semi-automatically take html text and convert it into a template. It took me over a day to try and put something together... and I realized that I still hadn't fully understand the use cases of the html templates, the possible forms of the html templates and just as important, the form of the templates can have small variations that could cause the code to not work.

So here I was, spending a god chunk of a day trying to write code to optimize a process that I hadn't fully understood yet. And I have a feeling that some of the biggest failures of software projects is a poor understanding of the use cases which may need extensive revisions to deal with unforeseen problems.

More important than making it work quickly, is that it works first. I ought to stick to that before putting effort into speeding up certain processes. Once I get this templating engine up and running, it'll be interesting to see what uses I can come up with for this code.

Building a web craling platform

I've been doing some web-crawling over the last few years. I started off with some really rudimentary pattern matching stuff but over the last 2~3 years I've become much more comfortable with text processing. To the point where I am becoming able to programmatically to edit chunks of text that I write. The power of really good text processing is amazing, especially when it comes to structured textual information.

I have spent an innumerable amount of time trying to figure out the best way of writing programs to parse text and I have been thinking of programmatic methods of getting the data out the information out there. But the more and more time one spends with text, patterns and strategies arise that can be used over and over again to get at the information embedded in text. What I want to be able to do is to create a simple frame work that will allow me to quickly create parsers for whatever text document I want to get at and keep a library of them so I can stream data from a variety of websites. Eventually, I might be able to make it easy enough that even non-programmers can be able to write parsers too and that may have some interesting applications.

Running 4 km (almost) every weekday

As a result of the commute, I've been working on getting the most out of my time. Getting home by about 8~9 pm everyday means that I don't have a lot of time to do much else. Considering that I am now making time to study for my CFA exam in December and trying to exercise everyday, that doesn't leave me with much time to do anything else after getting home.

I've noticed that I've put on a little weight, about 3 kg compared to last year. I attribute that to getting a big bag of almond chocolates from Costco and having a kitchen where I am cooking much meatier dinners as of recent. So I am needing some cardiovascular exercises to act against the recent bad diet (which I will get back into making more healthy).

I have a transfer stop on the way home where I need to change trains which I need to ride for a single stop before getting home. I've decided to take this opportunity to skip taking the transfer and jogging the rest of the distance home. I've been cycling on and off since the last year, usually doing long treks on the weekend (about 30 km) and I was quite pleasantly surprised to find that exercise on the bike transferred quite nicely to running as running 4 km. Progressing through the first weekhop of running nearly everyday, I have found that my stamina has increased and that I can push up my speed significantly. There are also daily variations on my ability to run, however I am feeling an improving trend. I should start timing my runs to see how I progress.

By the time I get home after a quick hop into the shower, I've got time to pump some iron using the weight training machine I have in my room. I also have some free weights in my room that I use from time to time and I've noticed that using free weights are great for full body workouts instead of working on isolated muscles. I still am working on trying to figure out the perfect routine to really push my muscles. It looks like I'll either have to hit a gym sometimes and talk to a trainer or consult some books.

Conquering Excel Macros

VBA is a terrible language. It's archaic and idiosyncratic. I recently got back into programming in VBA since yesterday to help out a friend at work to process data faster. It would take him weeks to what this program will be able to do in minutes. The only kicker is that it costs me time to write this code.

I've transitioned over to Python as my main programming language about 4 years ago and I've learned a lot from it since starting. Functional programming and the map, reduce strategy to computing data. Basically, if you can write out a function to process one block of data, it's just a matter of looping through the rest of the data array to calculate everything.

What python (and several other languages)  do is obsolete the loop and make them implicit. If you have a function and you have an array, then there is way to apply the function to all the elements of an array with one line of code. None of that

• let's create counter,
• create the loop structure,
• create an output array,
• pass the data to the function,
• dump the output data into the array,
• and increment the counter to get to the next piece of data

process. You have a spoon and there are many buckets of ice cream, what else do you need to know? Functions and list comprehensions work just like that. In a single line of code that 6 step process is gone. Code that can comprehend lists are amazingly short and the hard thing to do it limited to writing the function.

I work in Python and I think in Python when I program. Then I come back to VBA, it doesn't have the syntax to do that. What usually took me 1 short line of code takes me 4~7 lines of longer code to do the thing. It's a waste of time and prone to error.

So basically, I've spent the last 2 days making VBA more Python like by creating the data structures to alleviate the lower level management of data array comprehension to the point where I can throw arrays around and process them in 2~3 lines of short code. It's not a perfect solution, but much better and far more manageable. Compared to the old procedural code that I used to write, managing old procedural code is a nightmare to maintain. The great thing about list  comprehension is that you can worry less about the state of the program because the code becomes stateless. None of that "what was the value of the counter?" and that pizzaz.

There are times when the state of the program is important however, like for instance a user application: What screen am I on, or what settings have I changed, or am I still connected to the internet or something like that. But still, the overhead of dealing with states associated with data greatly decreases with list comprehensions (there will be cases where the data will require state changes in a function, but the overhead is greatly reduced).

At the time, instead of having multiple lines to describe a process, you can have just one which is far easier to understand because the unimportant scaffolding is hidden; the only thing showing are the important parameters and the name of the function. That is how good code is written. So farewell to the terrible looping structure, I've gotten rid of most of them and now left with the descriptive code that tells me what it does and what it operates on.

That's the beauty of higher level computing languages, in that you're able to do a lot with saying a little. There are even higher level ones where you can define your own keywords and syntax to them. A not very well known language is Lisp that allows one to do just that. I have no idea what is possible when one is able to define one's own language to suit whatever problem they are tacking with but I am quite sure it would be a very fascinating adventure. I've already seen my programming skills improve greatly by using Python. I can only imagine what else is there to learn by learning higher level languages.

The mind is a lot fresher after spending time away

I have found that I need to take 2 passes at a problem before getting it right and usually the second pass needs to be done after spending a day away from the task. My mind feels significantly fresher after spending time away from work and then coming back to a task. Things that I didn't notice before just jumps out at me, where as in other cases, I could look over something repeatedly and still overlook details.

I find this to be true to both writing and even programming. The first post of anything I make tends to filled with errors and even after double checking, I still tend to miss a lot of small mistakes. It's likely related to a bad habit of wanting to get something done as fast as possible and sometimes I find double-checking as a tedious task that gets in the way. One hitting the submit button, that feeling of needing to have something done as quickly as possible usually subsides and I am able to look at past work with a more critical eye.

There are times when I have been coding to find myself coming back to the code wondering to myself "what the hell was I thinking" when a much simpler solution exists. This is especially true when I make the mistake of designing a complex solution and thinking of the solution as clever.

For example, I managed to write a text parser with the feature that it could automatically detect if a text string was an integer, a floating point number, exponential or a date and automatically call the correct function to parse the data. The function was also extensible in that it could also be updated on the fly to auto-detect and parse other values.

I thought the smart thing to do was to create 2 functions, 1 to detect if a text string was of a certain type (returning true or false) and then a second function to perform the conversion and I had to make an elaborate system to keep the testing function and the parsing function paired together which I thought was kind of unweildy, but I managed to do it. A day later, I realized that I could just have 1 function that would either return a value or nothing depending if the parse failed or passed and used that as the indicator if the right function had been called.

I could have been bashing at a problem through an entire day with an inelegant solution... and I deplore inelegant solutions because why work hard on a bad solution when you could be spending time on a better one instead? Thinking like this is both a blessing and a curse because the resuts of my work varies between "really good" or "none."

Anyways, hopefully I'll be getting back to posting on a more regular basis. I have more pockets of time where I am able to think compared to before and I hope for this to continue.

Less time online and improved concentration

I've been getting back in to studying for the CFA exam and one thing I noticed is how terrible my concentration was. Spending one's time reading online articles vs reading books is quite a different experience-- a large amount of information has been accumulated into a single place instead of scattered through a variety of webpages that one would have to usually have to search through. Having a good table of contents is also a boon to immediately pinpoint where in a document that you would need to go to get the information needed.

By having a large volume of high signal-to-noise information available, it significantly cuts down on the time required to search for information (which I think of as a distraction). The resulting effect is that having good books to do research from actually helps improves one's concentration instead of being distracted by looking for information (which may or may not be relevant).

One other interesting thing I've noticed about myself is that I tend to focus way better when I am working with other smart people. I believe that having the synergy of having a few smart people around to cover for information gaps or thought gaps cuts down distractions by the need to look for information. I believe that addictiveness of, say, video games in general can be attributed to having all of the information a player might need very accessible through a very intuitive interface or by having an environment where one can immediately figure things out with simple tests.

Right now, I think that the internet isn't an ideal place to learn in depth topics through websites as either the quality isn't there, finding good quality content is simply too much work or the information that you're looking for isn't covered in significant enough depth. I think that books and other resources fill those gaps.

One project that I would like to work on is to start creating a repository/network of high signal-to-noise sources and will be looking into a variety of tools to help me do that.

Cutting down on wasted time

I've been working on cutting down wasted time recently, considering that now with the commute that I leave the home at 7:30 am and don't get home till past 8:30 pm, I have come to the realization that I have little time to myself. It is paramount that I make the best of it so I've been working on cutting down the number of distractions and time killers.

One of the best moves I've made is killing the habit of flipping on the computer immediately after getting home. I don't have a TV and have been living without one for the past 5 years without one and to take its place, the computer has become my media center. The problem is that the way I am spending time on the computer is just as bad as the way I used to spend on TV-- aimlessly doing nothing.

Instead of flipping on the computer to aimless browse at things, am becoming more stringent with my time allocation. First thing I do after getting home is either cook or exercise. I've also started making a new habit of not riding the train all the way home and getting off at 1 stop before I get home. Actually, it's a transfer stop and instead of spending time waiting for my transfer, I just hit the street and jog the rest of the 3 km home. I've managed to do the jog in about 15~16 mins and it's turning into a pretty good exercise routine.

Running this distance is actually quite significant when it comes to reducing body fat as I burn about 200 calories. One average meal for me is about 500~600 calories so I am burning off the equivalent of 33%~40% of a single meal. Assuming that I do this 3~4 times a week (ie when it doesn't rain) the reduction in the amount of calories the body absorbs is significant in addition to the health benefits of jogging.

I've also allocated about 1 hour almost every night to studying for the CFA exam (at a rate of about 20 pages/day) and have made this a prerequisite to turning on the computer. It's been about 3 years since the last course I took in university and it's rather refreshing to start studying again. One of the great things about reading study material is the much higher signal-to-noise ratio compared to reading articles on the net. The quality of information in books is far superior to what is published on the net. I've also noticed that as much as I like digitized information, I find that I like writing out my notes; there is simply a freedom in penning out notes that isn't available compared to typing out information (writing diagrams and sketching arrows is still ridiculously slow). Now that I think about it, if there was a great way to pen notes digitally and organize them, that would be awesome because I am one of those people that can easily generate volumes of notes.

I wonder if there are good note taking software where you could both type and use a stylus to sketch in other information. Like arrows, lines and even equations.

Once getting into a groove of "getting things done in rapid succession," I've found that a momentum kicks in. In the sense that I am far more likely to want to move on to the next item on my mind that needs to get done, without hesitation. Looking back at past behavior, the biggest hindrance to action is thinking about all the things that I ought to be doing and not knowing which of the things I should just simply do, because I have a tendency of worrying about "is the thing I am doing the right thing to be doing?" I've moved on to killing that by coming to the realization that time is scarce and doing something is better doing nothing. I've become far more effecting at using my time when getting at home, far more compare to when I was living closer to work and I find that ironic.

A change in environment is a good thing.

Working on my first patent

I've been charged with the design of some new optical systems based on geometry and varied materials to improve the brightness of light emitting devices. One of the great things about doing simulation work is that I can create 3D models, change materials and configurations faster than it would take the people fabricating the devices to create and test the devices built, because semiconductor fabrication is tough work.

Meaning that I can likely iterate through far more designs and variations than the processing people can go through in the same amount of time, probably by factor of x5 ~ x10 easy.

As a result of the work I am currently doing, I've been able to write a library that allows me to quickly generate 3D models, simulate them and analize the data faster than the guy that taught me to use the software (and has about 2 years up on me in experience using the simulation program). We tried to bring in another guy from a different department to do simulation work, but after 6 months of work, he eventually burned out... but I digress...

Now after getting out of the clean-room and having reasonable amounts of time to spend thinking and analyzing information, I have already found ways of increading output of our devices between 20%~50% with out standard models and I have identified a possible way of pushing that up to the 60%-90% range with some designs.While I am at it, I figured that it would be very cool to put out a patent to have something under my name.

Applying for patents is an expensive process with fees for the application and additional fees to maintain the patent over it's period. As an individual, it might be prohivitive to do this kind of stuff, but if you're working for a large corporation that can do this, why not?

Unfortunately, the patent review process takes approximately 3 years to go through, but I figured that the earlier I get started, the better. I just need to get it through the internal review process and find time to talk to the legal department here... this is going to be interesting.

So many thing to learn, so little time

One good thing about moving out from where I live now is that I value time much more compared to what I used to previously. Meaning that despite having less time, I am aiming to do more with it.

The current things I have on the chopping block are the following:

1. Take and pass the level 1 CFA exam in the winter
2. Work out 5 out of 7 days during the week for at least 30 mins
3. Pick up emacs (for text editing and programming)
4. Learn Ruby on Rails to understand how to develop web applications
5. Dedicate atleast 30 mins/day to learning Japanese for the JLPT exam in the winter

Time is incredibly scarce right now and I am pretty much at the point of even using small smidgens of time at work to help myself accomplish these goals. Just glad that I can make time to do things at work instead of doing some manual labour job.

Living with the commute and etc

It's been about a month and a few weeks since making the move from the company dorms. I pay a higher rent now but having my own place where I can invite people over is nice. I now live about 20 minutes outside of Tokyo, instead of 1 hour. The trade off, however, is that it now takes me an hour to get to work instead of the 10 minutes I used to spend before. The time delta of 50 minutes times 2 is actually quite significant. The slate has an article on commuting entitled "your commute is killing you," explaining that the further away you live, the more your health, social life, productivity and marriage suffers. Fairly obvious stuff, considering that you have less time to do everything else. 2 hours out of 16 waking hours spent on commuting is more than a 10% of productive time, which is painful.

On the flip side however, I do enjoy living closer to Tokyo and it has made getting out into the city on the weekends far more convenient instead of the 1 hour ride if I wanted to get out there. I can spend more time with friends compared to what I used to be able to do before. Just as important, however, is my ability to network with people in Tokyo will improve as Tokyo is far more accessible to me.

What I'll be looking forward to doing over the course of the next year is to get acquainted with the hacker community in Tokyo, hopefully find some intelligent, driven people to work on some projects and bounce around ideas. I've already found the Tokyo Hacker space where they do meet ups and work on projects and also a Tokyo Startup Weekend event which happened last week. Unfortunately I wasn't aware about it and it ended just a few weeks ago but I'll be looking to become more involved with the entrepreneurial communities here to find cool people to do stuff with.

I've been meeting lots of people recently, but there aren't a lot of people that are interested in doing business or have a business like mind so I'll be spending this year searching for entrepreneurial communities.

Am alive, just been busy moving

Completed the move to the new place just last month. Will have some more details about it in a bit. Been quite busy getting the furniture and unpacked. Normal semi regular posting will resume in a bit.

Financial wealth vs physical wealth

Been reading about the commodity bubble and crash of 2008. It is interesting to try to understand how the motion of money that causes the commodity bubble. At first, I was thinking that money can move into a commodity class (as in be stored) but in reality, someone is exchanging a commodity for that money, which breaks that model. I wanted to model a change in price as a function of the amount of money that goes into an equity, but that doesn't seem to be the case. What basically happens is that the money is exchanged for a commodity or equity and the transaction ends there.

The value associated with an equity is the result of it's instantaneous trading price and there is a disconnect when the value of the equity is tied directly into the "real" money supply and the physical commodity.

Assume a closed system of $1 billion dollars and 100,000,000 tonnes of copper:

• Let the price of copper be $10/tonne, the total value of this market is $2 Billion
• Let the price of copper be $30/tonne, the total value of this market is $4 Billion

Just because the price of copper was more expensive in the second case at $30/tonne vs $10/tonne does not necessairly mean that one economy is more "rich" when the amount of physical resources remain the same. I would be much happier to if I (and others could) a good $100,000 house versus a terrible $500,000 bad (top of the line) one.

I think that there is a gap if the measurement of "wealth" is "money."

The balance of living in a one room apartment

Another 3 days to move into the apartment and packing is underway. The number of belongings are less than expected; I regret not haggling with the moving company for an even cheaper price. I looked up the weather this weekend and there is a 50% chance of rain. I've never had to move in the rain before and it will be interesting to see how the moving company deals with it.

I did some more research last weekend and found out that renting a truck for a day was 8000 yen compared to the moving fee of 40,000 yen that I am paying. The only catch is though my driver’s license in Japan allows me to drive stick shift, I have little experience at it (perhaps 2~3 times). I would be interested in renting a manual car and learning stick because saving 30,000 yen for a single day's work is worth the effort.

I spent the last weekend looking at furniture for the new room as it will be a much bigger compared to what I have now. I can fit a sofa, bed, a desk and my weight training machine and the question is, do I really want to buy so many things? One of the things that I really liked about living light on belongings is that it is easy for move around (though I haven't been taking advantage of that).

Though it has been tempting to buy furniture in advance and have everything arrive on the same day, I have decided that it would be better to live in an empty space first to get a feel for the room before loading it up with stuff; I would rather have bought things to realize that it doesn't fit in well with the room. I'll be pretty happy with moving as it'll be really nice to be able to invite friends over.

On preparing to move

There are hidden costs (and benefits) to everything. I've been living in the company dorms for the past 3 years and by virtue of the age limit I am moving out by the end of the week to my own place. Leaving is a blessing because despite the savings in rent ($200 monthly), I probably have incurred plenty of social costs by living in the dorms. As a result of living in the company dorms, I am far away from Tokyo and can't invite guests over because of the building rules. As a result, I only see friends when invited to an event or when I host something elsewhere.

One part to living is the social aspects of living is knowing people and being able to do things/interact with them. The living conditions up until now have prevented me from doing as much as I would like to have done. Though in retrospect, I did end up doing more traveling around Japan from the savings from really cheap rent.

I have managed to find an apartment just outside of Tokyo, a brand new building and I will be the first tenant into my apartment. It is a fairly spacious 1 room apartment with an insulated floor and I have managed to select a corner room, where to one side is a wall shared with the outside and the other wall is shared with an elevator, meaning that I have free reign to play music and watch movies as loud as I would like and am incredibly happy about this.

I face one other dilemma; in that the room is really spacious and I would like to buy some nice furniture go into where I live. The problem I fear is having too many large belongings that eventually tie me down to a place.

The best solution I can come up with so far is to make more money to make the costs of buying the furniture insignificant to my income, making what I buy "disposable."

 

The post earthquake situation in Tokyo

For the large part, life in Tokyo has largely returned to normal. There are still some noticeable after effects of the earthquake still lingering in the city however. Since the shutdown of the Fukushima reactors, the entire Kanto region (the region surrounding Tokyo) is running at reduced power capacity and the entire region has managed to cut power consumption by at least 25% since the earthquake, alleviating the need for rolling blackouts through the region.

A link to the power charts for the Kanto region is provided here. The blue dotted line represent's last year's power consumption, the small blue bar is the total peak power available and the pinkish line represents yesterday's power consumption and the blue bars represents power consumption for the for the current day.

It's rather fascinating to see how much more power people and companies are able to cut and still maintain fairly "normal" operation. I doubt that you will see this level of power consumption elsewhere in the world however, as the attention to detail and the character to voluntarily reduce consumption is the same with other countries. The above graph could be a representation of the amount of "disposable power" Japan has.

In more detail, office buildings have reduced power to lighting, reduced the number of elevators and escalators operating. The trains have turned off the air conditioning and turned off lights inside the cabins during the days (it gets incredibly dark when going through short tunnels and really shaded regions). I expect power consumption to slowly rise as industrial regions will slowly rise as companies start brining more equipment and facilities online.

We have yet to determine if the power shortages will continue through the summer as one additional power plant is scheduled (since before the earthquake) to be taken offline for inspection through the summer, possibly bringing back the blackouts. There have been talks through the news that this might be not necessary as we have been seeing unprecedented levels of power conservation.

Supermarkets are currently short on food products as the shelves for certain products are either empty or only half full; coupling that with the current dim lightning resulting from power conservation, it presents a rather dreary atmosphere. The supermarkets are short on certain vegetables, milk, soy bean products and bottled water (which were quickly snapped up because of fears of water contamination).

The schedule for the current project that I am working on is being pushed back by about 6 months (and we are really rushing it at this rate). I expect that much production in Japan be significantly reduced and research delayed. Since the currency jump after the earthquake, I have decided to move some of my funds out of yen and into other currencies as I expect the yen to weaken throughout the year. Already in less than a month, the yen has dropped significantly.

I expect the earthquake will send slow shockwaves throughout many technology companies throughout the world as Japan produces many high tech components in many electronic devices. This might be an opportunity for other emerging market players to pick up the slack for reduced production from Japan. Right now, this probably is a very big opportunity for other Asian countries like Korea and China to play technological catch up with Japan. I expect competition to be fierce once Japan gets back on its feet.

Macbook

So I managed to get my hands on a macbook recently and started learning how to use it. There are obvious differences when it comes to using one of these machines compared to a Windows or Linux machine. One thing that drives me up the wall is that I still haven't figured out how folder navigation works other than double clicking on icons to open them; the standard "hitting enter" on files or folders doesn't work here.

There are other notable differences; for example, the function (F1~F12 keys) are actually mapped primairly to notebook functions (like screen brightness, volume control and etc) and you have to use a function key to activate their "normal" behaviors (ie function-F5 for refresh). There are, of course more things that I need to get used to, like how applications are installed and removed (things work slightly differently compared to usual Linux machines) so it'll take some time to figure those things out.

I still remember my first venture into the linux world after buying the first version of the EEE-PC. It took a good week of tinkering to really figure out how to do software installation and figure out what packages I needed for things like multi-lingual support (I also ended up killing the OS a few times by deleting the wrong things too). I suspect that I'll probably fall through similar pitfalls while learning how to use the mac.

One thing that I do like about using the mac, however is the polished multimedia editing tools, which are leaps and bounds superior compared to the open source alternatives. I guess when it comes to software that you pay for, it is what you get.

Interms of the interface, I like it because it has a cleaner look compared to the usual windows interface and the applications look pretty good. Still haven't had a lot of time to tinker with things but will be looking forward to playing around with the computer some more when I have time. After pretty much graduating from playing games on Windows, there really isn't much of a reason to stick around to a single platform anymore.

The effects of radiation being felt in Tokyo

A friend of mine posted that the "air tasted like metal," though unaware of it, those are signs of elevated radiation. The government had reported elevated levels (though not physically damaging [yeah right?]) of radiation in Tokyo this afternoon. Work has been cancelled for tomorrow and the day after so I am making plans to relocate out of Tokyo for at least until this weekend.

CNN's coverage of the Earthquate is crapy

I generally abhor news reporting just because they always focus on disaster, scandal and damage all the time. Maybe it is an American thing, but for the first time in a long while, I am currently at a friend's house watching the news with a Japanese news feed and CNN's news feed side by side. And I have to say, I am in awe of how shoddy CNN's reporting is. Nothing but the same clips of the Fukushima explosion over and over again, exceptionally dumbed down explanation (read, elementary level?) explanation of a nuclear power plant and more. I am surprised that they do not have any bilingual reporters translating and reporting on the direct press conferences from the Japanese government.

Though the Japanese news stations, through do replay some clips of the damage out here, I am far more aware of the evacuation process and the condition of people around the Sendai area; which I think is far more important compared to just talking about the "massive damage" from the earthquake.

News organizations have too much power over the way people form their opinions about the world and I don't think they are good enough to handle that kind of responsibility.

Backing up data as a precaution

Going to be leaving the home for a bit to meet up with some family. Decided to backup some critical data to the cloud, just incase I won't be able to return. In disaster, the internet is incredibly handy.

More info from a friend in Sendai

More people are coming online on Facebook reporting that they are ok. Seems like shelters are taking care of people there. One friend was able to cook several days worth of food. My friend reports that there is no power or electricity. Seems like power is up and down depending on location.

People tweeting real time on the news

Been watching the news online and there is an integrated tweet feed. Most tweets are pretty useless and I wish there was a way to block them.

Note to self: look at sources of reliable and unreliable tweets.

From the Sendai City webpage

For those that are interested. Sendai city home page is up:

http://www.city.sendai.jp/

-Bus service is intermittent in some places, operational in others
-Hospitals accepting patients in severe conditions
-People are recommended to stay home
-Natural gas service is intermittent
-People without water can have access through sources at schools
-Garbage collection not in service

Hitting the supermarket in the morning was smart

The super markets and convienance stores were busy today. People have been stocking up on food since the quake. I now have about 1 weeks worth of food cooked and prepared. Other friends have mentioned that some stores are empty of food that can be immediately consumed.

Japanese reporters on radiation levels and evacuation distances

Looks like the reporters are seeding panic by wondering if the evacuations distances are sufficient. They are worried about the government covering up information and if the evacuation distances are sufficient. Exceptionally pissed off at the talking heads that they aren't doing any research.

I wish there was some kind of market to force them into a bet... where I can bet against them.

Sendai Status via Twitter

For those that know Japanese:

Via Tetu on Twitter: http://twitter.com/mahhya94/status/46425985838620672

青葉区五橋の姉と連絡つきました。13：22現在 避難はしていない様子。街中は建物もそんなに壊れてないと。 30分前13時ごろ電気復旧。水も復旧

Translation:

Go in touch with sister in Aobaku-Itsutsubashi. As of 13:22 does not seem to be evacuated. Buildings do not seem to be that damaged. At about 13:00, power restored, water running

Via Tetu on twitter: http://twitter.com/tetumemo/status/46476055724298240

【追加情報仙台市の友人から】仙台市役所前のみ電力供給あり、取引先の会社へ避難中。　長町南から長町駅、商店街付近は建物の倒壊は無いものの道路がうねり、交通が困難

Translation:

(Further information from friend in Sendai)Power only available in front of city hall, taken refuge in company building. At south Naga-machi, objects on streets not from fallen buildings on streets. Traffic is hard.

I've heard further reports that water isn't running in some areas...

Fukushima Nuclear plant explosion:

Calculations on the radiation levels:
Current reported dosage levels @ 1000*normal.
Normal radiation dosage @ 2.4 mSv/year
need 1~2 Sv dosage to start to feel nausea or vomiting -> time to that occurs is about 5 ~10 months assuming that you are *at* the power plant.

Dosage levels will probably drop in an inverse squared power law away from the power plant. Meaning we've got plenty of time to react.

Sources:

Radiation rate @ 1000 x normal: http://bit.ly/eGht2S
Radiation poisoning levels: http://bit.ly/bRheP6

Quake notes

Random news articles:

• Fukushima Evacuation
• Japanese Strict Building codes
• Quake warning system

News Streams:

• TBS TV
• NHK TV
• Al Jazerra

Facebook, faster in disasters

Spent most of the evening and night yesterday on Facebook reading from friends reporting in that their families are ok in Sendai. SMS and cell phone contact is really slow and I thought it was ironic that it was faster for me to Skype home and contact people through facebook. I'll make a mental note of that for the future. So much information just came so fast that I hadn't the chance to organize it into a post... the damage in Sendai is extensive and we are still having aftershocks.

I think I am going to take it easy today... wonder if the grocery stores still have food.

Earthquake, I am ok

Incredibly sad the earthquake hit Sendai. I've been to some of the places they've shown on TV with damage. I am ok at least. Lots of information going through Facebook right now. Checking with friends in Sendai to see if they are ok.

Thinking about the Monkey Portfolio

In the old days of high school we used to have multiple choice scantron tests and we used to call the monkey score the score you would get if you just randomly started choosing answers. We used to have 5 possible choices on these tests so the monkey score would be 20%.

I was having a conversation with a friend recently about financial planning and the guy is a financial planner actually. During the discussion my friend proclaimed to me that he could structure a portfolio for clients with different risk characteristics with averaged annual returns ranging between 6%~12% and that he would recommend a 12% portfolio for young people for fast growth. His reasoning behind that was that you could take any 10 year window of the stock market and find that the market compounds at an average of 10%. Since he worked at a financial institution, they had better tools to get better returns which warranted that they could get that 12%. I decided to take his statements to task to really see if this was the case... unfortunately it wasn't.

To be generous, I decided to take a nearly 30 year span of the stock market starting in June 1981 until Feb 2011 (chosen at random) to calculate compounding rate. As of June 6 1981 the Dow Jones index was at 976.88 compared to Feb 1 2011 value of 12130.45, which is a 12.4x increase. Calulating for the compounding interest rate, all I would have to do it take the root of this between the time span to find that the compounding rate to be approximately 8.86% over a 30 year period. The result is striking to me that a 10% compounding rate is optimistic at best and 12% is a bit out there.

The Dow Jones Index since June 1981 to Feb 2011

So next, I decided to do take this a step further and assumed that I invested into the market at a constant rate of $1000/month for whatever the price the Dow Jones Index was going for until the beginning of Feb 2011, what would the annual compounding rate be after ~ 30 years? The result is illustrated in the graph below:

 

Compounding rate calculated while making constant contributions

Sure there is a lot of movement in the compounding rate in the initial few months but as time slogs along the compounding rate begins to stabilize at 6% but drops down to 4.7% after the recent economic crash. From the results we are nowhere close to achieving the 10% compounding rate that was initially proposed. Just as a note, the graph bottom axis is in months but has the same time span as the above Dow Jones plot, I was just too lazy to change the axis.

The gains however are still significant even at a lower compounding rate though, assuming a $1000 contribution into the fun over 356 months, the total contribution turns out to be $357k (assuming +$1000 at month 0) with a portfolio value of  $1.38 million after nearly 30 years of constant investing. Sure the market might have had compound growth in the upper 8% range, but the dollar cost averaged purchases of stocks over the past 30 years tells a different story. The guy was obviously making a pitch at me and setting reference points is a common sales tactic, but the reality of the matter is that a 10% continual compounding rate is optimistic. When it comes to financial advice and products, keep your wits about you.

 

Some notes on web programming

It seems to be taking forever for my web domain to get registered. After registering for a webhost on Sunday, 2 days have passed and my application to get a web domain is still pending. Not particularly happy about that but there isn't much else I can do except wait, though 2 days seems like an eternity to me.

While doing the waiting, I am looking into the following things:

• How to setup an Apache server and get CGI includes for Python working
• How to create interactive applications for webpages
• How to manage data using a SQL type database
• How to create web applications on infrastructure on the Google App Engine or Amazon's EC2
• How to program websites with AJAX (since I don't like web pages reloading on me after doing a query)
• How to use Python to generate html templates or generate CSS templates through Python
• How to create menus, buttons and images through programs since I am too lazy to use things like Photoshop to do tasks like these manually

I have a feeling that we are going to see a trend away from desktop data management happening as online servers will be dealing with data management and home computers will become more like clients in the future.

I need to certify myself as fluent in Japanese

Learning a second language is a pain in the ass, especially when you have access to internet in English. It's a terrible habit of mine, that I exist in a bubble of English and use my Japanese only when necessary. For being here for 5 years, I should be able to do better.

I've also been looking at new job opportunities, people that speak both English and Japanese fluently have a leg up, there are lots of interesting opportunities out here to work in a multinational firm or setup new businesses, especially when foreign companies need people that understand the culture and want to setup base here. I have had my nose buried too deep in work and haven't been working on my Japanese as seriously as I should. My bad.

I want to raise my salary by 25~50% by the end of this year by improving my Japanese skills and picking up some marketable skills for use in different environments. One thing I have learned about working in a corporate environment is that you'll get nowhere if you're a bad communicator.

Foray into web publishing

I've finally broken down and bought an account at a webhost today. The site I chose was iPage, which allowed me to register with them, get a free domain name and hosting for a year for $54. Not bad considering that I now want to learn the basics of hosting web content on the net. I've done this kind of stuff before, but on my personal machine in the past. If I am to create more intricate web pages and possibly applications, I should learn how to host them on the internet, which means learning how to use a webhost.

I made one mistake with registering with these guys however, the problem being that they do not have SSH access into their servers, meaning that I cannot run command line related stuff to check the version of the software they have, but they seem to have a large repository of interesting packages to play around with. Later on, I may see about using Google's app engine or Amazon's EC2 frameworks for scalable software. It will take me time to learn things but I think there is interesting work to be done in this field. This blog may or may not move, but I am in the process of registering my own domain name and interested in developing my own web presence.

Contemplating getting a macbook

The last time I was in Canada, I was back home to be the best-man for a friend's wedding. My dad, being a Mac convert had a few Macs lying around the house and I found out that their iMovie software was pretty damn good and made a slideshow movie through using the software. I was thoroughly impressed with the bundle of good media software with the Mac. It is as if the Macs were specifically designed for media kind of work. I never thought twice about getting a Mac since I switched over to Ubuntu-Linux, which is a packaged version of Linux. For free, it's pretty damn good and has nearly all of the basic functionality I need.

I've noticed, recently there are some rough edges that need some smoothing out, mainly in the section of professional software. The great thing about linux software is that it is free and they work pretty well-- I can pretty much do nearly just as much, if not more on a linux machine compared to a windows box, for free and even better. I really like how fast Ubuntu boots up, how quickly I can log in and how responsive the machine is even right after logging in (there isn't very heavy background software loading after you log in). It's pretty tight software. The only thing that is lacking, perhaps is a really good video editing suite or media management editing suite. The one thing that I liked about Apple software is the level of polish that goes into the software available for the Macs-- they simply work and usually are quite cutting edge.

Compared with the Microsoft ecosystem of software, I've found that looking for windows based software on the internet to be rather spammy and sometimes the software sites on the net can sometimes even seem a little dubious. Perhaps it is a result of a smaller and tighter-knit Apple perhaps? I am not entirely sure, but we are starting to see something similar happening to the iPhone/iTouch app store where sometimes dubious software sometimes does pop up (though putting software up there is done through an approval process which has wrought ire from some people).

Anyways, there is a bit of a premium when it comes to buying Mac hardware, but since the base of their software is Linux based, it shouldn't be too hard for me to pick up. The question is if I want to start off with a laptop or just an iMac mini (which I think it kind of pricy at $799!).

Keep a spare keyboard

The keyboard managed to die out on me while doing some administrative stuff on the computer. I've been working on a wireless keyboard and mouse since the summer and have been really enjoying it. Sure the battery dies once every 3 months but it wasn't much of a big deal, until the keyboard died on me that is.

After wondering if the keyboard was in a bad position, the receiver was in a strange location and wondering if the battery was dead, I have pronounced the keyboard dead. Generally, indicator LEDs aren't standard with wireless keyboards because they shorten battery life, making debugging rather tricky. I eventually had to dig out an cabled keyboard to connect to the desktop... only to find that the keys were rather sticky. Then it struck me... I spilled some really sticky liquor on the keyboard since the last time I used it and quickly took it to the shower where I doused it in hot water.

Suffice to say, I won't be able to use my desktop computer today and am currently using the internet through my netbook. I've put an order for a new wireless keyboard online and the total cost for a new one is about $25 bucks, including free shipping. Not bad, it's something that I can live with.

Sometimes, you have to entice people to commit

Working alone is tough, there is only so much memory, attention and knowledge that a single person has. Not to mention that the cutting edge of technology is a moving target-- what you don't release this year might be outdated by the next. If you want to make a killer product of service, you have to make something good and do it quickly. My best analogy to this would be like learning to ski well-- learn to lean forward, even to the point that you think you might do a barrel roll, do that.... and lean forward some more.

I've been in a recruitment phase for finding a community of smart people to work on projects with and eventually turn those into a viable service or product and there are great opportunities out there for data mining/business intelligence type applications. There is so much on the internet that we have just begun to scratch the surface, but people know that the market is there and there are many startups that are moving into this realm.

There is only so much that I can do on my own and that has been one of the biggest realizations for me. Yes, it is true that I could probably learn what I needed to know to do something, but it takes time to be a master at sometime. If you truly want to do something big, you're going to need a team of people to do it, a tight group of smart people.

I've been making some programmer friends recently and I've made several pitches to try and get a project off the ground. Yes, I can convince them that some of my ideas are interesting, I also enjoy bouncing ideas to find new avenues of use. I have some code written and some proof of concept, but I need to learn how to make people to commit. I tend to generally be soft on people, like when I tell them "spare whatever time you have" or "get things done whenever you can," to be honest going easy on people like that is a mistake and you'll get no real commitment from them; unless of course, that the people you are talking to implicitly feel strongly that you are going to succeed with or without them (but still, it is a weak play).

If you want people to commit to working on a project with you, you do have to entice them. Show them that the project has momentum, that you are strong willed and believe that it is a good idea and that if they want to join up that they have a short but reasonable time to do so, because people usually don't do things unless there is a deadline to for decision; it is simply human nature (oddly enough, I think it is similar to dating and that you shouldn't give a person that you really like the opportunity to go out with you whenever they feel like it because you are implicitly saying that you don't value yourself that highly... but I digress).

In short, you have to act with urgency and find ways of making what you do a priority for other people if you want them to join up or else things will happen "whenever," which translates to "never."

Now a fan of audio books

I had an American friend when I was back at university in Sendai that loved his audio books. While working in the clean room and processing his devices, he'd be in his own world listening to whatever audio book in his iPod. I never really understood audio books back then because I read much faster compared to then I could listen and I would get irritated by listening at the pace of the speaker.

Fast forward 3 years, I started listening to a few audio books while exercising, riding my bike and while on the train. The lack of having a (sometimes) large, heavy book with me was definitely a good thing, because I had less heavy things to carry around with me and I'd be learning during times when I had nothing else to do, which means time productively spent.

An audio book can take about 6 hours to listen to (about 1 hour/chapter) but it isn't bad considering that for a normal commute to Tokyo takes about 1 hour one way meaning that I can finish 1/3 of a book just on a trip out to Tokyo and that doesn't include the time for me while walking around. The other nice thing about listening to audio books is that I find the information content from books to be much more nourishing as the topics covered in books are covered with greater depth compared to what you generally find on the internet. The only disadvantage for audio books is that books discussing complicated math or needs graphs to explain ideas becomes very hard to comprehend, so topics that do not require a lot of "rigor" are usually good to listen to. Hopefully I'll find some good Japanese audio books to further develop my Japanese skills.

Updating the resume

It's been a long while since the last time I updated the resume. The last time I had updated it was nearly 3 years ago, after graduating from my masters program. I am (un)offically in the hunt for a career adventure. I believe that I can earn more and grow faster than what the managers think that the average person can over here.

I mentioned earlier that I wasn't so hot about waiting 4~6 years for a promotion. I think that one of the biggest faults of an organization is limiting the speed at which a person can climb the ranks is a fatal mistake, since no one will have an incentive to take on more responsibility than they have right now. But hey, that's bureaucracy and politics for you.

Interestingly for the past 3~4 months, I have been looking at other opportunities in different organizations, the pay is dramatically better and pretty good for even a young person like me. I've started sending out applications... no real bites for positions just yet.

It's been a long while since I did any real work hunting, preferably, this time in a more "Western" organization as I find my mental capacities are much sharper when I work and communicate in English. Good Japanese skills here are very important and can even improve the value of a worker too and I intend to work on that as well... I just don't have an incentive to do anything at the place where I am currently employed. But I digress...

What I realized after getting into the working world again (since the last time I did any kind of job hunting when I was graduating from my undergrad studies) is that the nature of the resume has changed. I was good at marketing myself as a student with my resume, which was padded with a lot of good academic achievements, what I realize now is that I can't play those up as much any more and have had to chance the format of my resume to reflect the kind of work and my new work interests.

Not in all cases, can one send a cover letter out for each application as for the recruitment sites that I've been working through now only use the resume to evaluate a candidate, to which I realize my resume is too engineering heavy for the positions that I've been looking at.

The other thing about looking at job postings, especially at recruitment sites is that they have concrete job descriptions and approximate salary ranges for the kinds of people they are looking to hire, which is really different compared to the standard Japanese system where you just apply, then they accept you and then figure out where in the company they want to send you. I am far more comfortable with the normal western hiring practices.

I've made the mistake of sending out too poorly tailored resumes to too many job openings that there aren't any new good ones at the moment so I am going to have to slow down and think of strategies of improving my skills set to get a good position elsewhere. It is an interesting experience to be back in the work search world again.