Optimize for investments or salary

My time is limited and in my spare hours, I've been trying to write code to better understand the stock market. My current work involves understanding stock options, which are quite the beast. I've so far gotten as far as writing parsers to get information form the internet and have gone as far as running monte-carlo simulations based on a variety of models but the problem is that I really don't know if these these models are correct and the more I code to create models,  the more I realize the less I know about modelling the stock market.

There is just so much on my own that I will be able to figure out on my own... then I thought about buying some advanced books to read. Then it hit me, I've got a lot of learning to do and I started to wonder if my efforts in this field would be worthwhile.

It is now also the end of 2011, meaning that my expenses spreadsheet for the year is nearly complete and I have a good idea of how much I've spent and saved for the year. Since moving out of the company dorms and into my own apartment, housing costs for me have skyrocketed and the amount saved dropped by about 33% compared to last year. This is even accounting for increases in salary. But the thing is that doing more research in stock picking may increase my investment returns by a few more percentage points. This begs the question, would I be better off investing that time in developing better skills to raise my salary?

Thus, I've started doing more research into engineering salaries and pay for jobs that I could possibly do and for jobs around the world. Thanks to the exchange rate between the yen and the dollar, the numbers look pretty good, but this is after a lot of overtime pay and travel pay that I got this year. On a dollars/hour basis, I don't think that the hourly salary was all that good.

Been Busy

Been really busy the last month and a bit to do any real writing, but a lot has been going on and I haven't had much of a chance to do much writing as of late. I've been flying between Tokyo and Kagoshima nearly weekly to help out with setting up an inspection machine and writing code to speed up the processes. Also, getting paid travel expenses and a small stipend is marginally nice.

On another note, I got a better than usual work review...after all was said and done, however, the better review led to only about a $600 increase in my winter bonus. The software I wrote cut down processing time by about 5 days and gave us more time to analyze data to improve the success rate of the process. If ouput and pay were plotted, the graph would be very sub-linear for the place where I work.

More notes to follow at a later time.

Defining the problem actually takes up a lot of time

I've been flying between Tokyo and Kagoshima for the last month and a half after doing my first 2 month stint in west Japan. I've already manage to take a 1 week process down to 3 days and while allowing 2x more samples to be processed simultaneously, meaning 4x throughput.

By the time I got to Kagoshima, some of  the engineers have been working on figuring out how to get the process to work and determine an algorithm for sample inspection. Armed with the basic information from these guys, I started writing software through the night shift and my work spawned into a project of it's own.

I've already finished writing an application and trained a technician to take over the job. I've started working on the next version of the program to try and have the entire process automated since my program doesn't work in some special cases. Classic example of the 20-80 rule, where 20% of the effort gets 80% of the output.

I am treading into territory where other people haven't gone into with the data analysis and I've noticed that I am spending less and less time writing code and more time trying to better define the problem and the exceptions that my code needs to deal with before writing any code. It's somewhat frustrating to not be writing code to get things done but I guess it's part of the process.


It's odd how the motion of "doing" something is so closely related to progress, though planning and preparation is also a very important part of solving any sort of problem. Being busy for the sake of being busy is detrimental. I generally like disappearing from the office when working on a tough problem that I haven't managed to wrap my head around and go for a coffee. I find that I understand a problem well, I work infinitely faster because everything is in my head and laid out.

There is something called the "drunken sailor's walk" which is a classic problem that nearly all probability courses covers at some point. After n random steps, the furthest you've traveled away from the origin is related to the root on n. If you walk with direction, the distance traveled is n.

In other words, if you plan things out properly and minimize mistakes, you'd be better off by the factor of a square. If you wonder why there can be such a gap in skill or level in the world sometimes, I think that it is the expression of this phenomenon.

For those that have the time, I suggest reading Richard Hamming's book The Art of Doing Science and Engineering: Learning to Learn, which is a fairly ok read. There are some interesting notes from a well known researcher, from which I gleaned this idea from.

A disappointing experience at a computer store

I did some more thinking about putting together a server recently and came to the conclusion that even though I would be paying a little more money to run my own server at home, I could get close to 4x more performance and unlimited hard drive space instead of renting one. I also though it would be a good experience to learn to set one up should I need to set one up online using one of the services like Amazon's AWS.

I put down the cash for server parts last Friday and so far, the CPU, motherboard, case and memory arrived. Since I was going for a server system, I didn't need a monitor or a fancy video card for the system which brought the price of the computer down considerably. When going for the slim computer case, I made the mistake of purchasing one without a built in power supply. Thinking that getting a power supply wouldn't be hard enough, I dropped by the closest physical computer store  I could find online only to be incredibly disappointed-- all the power supplies were really expensive and the cheap ones that they were selling were "sold out".

Obviously, I wasn't happy with spending the 20 minute (one way) commute on my scooter to get to the store, spend time looking around to find out that they didn't have what I wanted at a more expensive price.

After getting home, the first thing I did was go online to the Japanese equivalent of "pricegrabber.com" to source a part and I was done with making the order in about 10 minutes. The only difference is that the part will probably arrive in 2 days instead of me having it immediately, but when it comes to being busy, waiting isn't a terrible thing.

Over the past few years, I have been doing more and more online shopping instead of going to stores to buy things instead. It's simply more convenient. I really wonder how much has internet shopping had impacted the retail business and the number of old school stores that have gone out of business as a result of it.

I also move around every 2~3 years making finding new places to buy things fairly important to me somewhat time consuming. Ordering online never ceases to fail me since I get what I want delivered to me quick enough. But back to the computer stores, I really don't understand how they can keep up a business where people have to spend time to go to a place to buy over priced product. The only one possible answer that I can think of is for people that aren't as computer literate to talk to people to get what they need. If that information, too, can be accessed in a manner easy enough online, I think we can kiss the big box computer retail stores goodbye.

If this then that

An interesting automation site, I thought that the idea was interesting when they went into open beta after Sept. I checked after 2 months to see how they were doing and found that they hadn't updated the blog in 2 months. I think they're dead in the water right now and the people that worked on the site are off working on other things.

From the stock market, business and life. Things take time to happen and you should make it a habit of checking old news. So many thing that were at the front of our mind one day become totally forgotten later. I don't know why the time span for ideas is so short, but it would be wise to periodically check old news from time to time instead of just watching the latest headlines; because just tracking the new and not tracking how things go is meaningless.

The Autobiography of Steve Jobs

I haven't read a book in a real long time. Importing books is expensive and I generally abhor having more things to clutter up my room. Then came my desire to tinker with the kindle for the PC and I found reading incredibly refreshing.

After about 3.5 days of reading on both the computer and mainly on my iPhone, I've finished the book. I read of the life of a great man. For once history will have something positive to put among heroes because there just haven't all that many in recent history.

As a person to emulate, I doubt that many people would want to be like him; but to every great man, there are key ingredients. I've turned 30 this year and work life practically ends at about 65. If you really want to make a bang at something, have kids and do everything else, there really isn't a whole lot of time. Every moment is precious and I am reminded of that after reading this book.

Never have I been so consumed with putting my mind to doing something in a long while. After reading the autobiography, you can tell it was put together with care, with countless interviews and decent writing. After finding myself so consumed with reading for the past few days, I just came to the realization at how much the internet sucks; as in "holy shit, it really sucks," because there is so much garbage out there.

Maybe I haven't been going to the right places for content but, I've gotten tired of fishing through the needles in the hay stacks of social sites looking for something good; just simply having a good read was so amazingly refreshing. Yes, I will admit there are some good reads on the internet. Some great essays out there even, but nothing as cohesive as a good book. I missed that, I really did.

The other great thing about reading about Steve Jobs is that he was incredibly intolerant of things that bothered him. I found myself to be quite the opposite recently, being as accommodating and accepting as possible that I noticed that I lost a bit of an edge to life. When you get to the point of accommodating to the sub optimal it just turns into a slippery slope of accepting worse and worse. Sometimes you just need to draw the line even if it means you might have to be an ass.

There is a lot about the book that I really enjoyed, but I feel that I would do no justice to even talk about the story because what really is the point about just saying if something was interesting or not? The act of simply talking about something just feels like it has no value to me and in it's own way is what was the story of Steve Jobs about-- making things happen.

To get good at the stock market, spend time away from it

The stock market right now is one big drama and generally it is usually the people that spend the most time watching dramas are the ones that get the most emotionally involved and start losing money. I made that mistake once before in 2008, just waiting for the perfect time to catch the bottom and doubled up on it because every motion up and down seemed huge to me and when the stock market dropped once in October, I thought "this is it" and it really was "it" for me with a gaping hole in my trading account.

Gone are those days and I've gotten better. To be honest, I think the market in general is on tilt, to use a poker term and it is probably best to leave the table as it is and take a drink while the players sort themselves out. Because really, when everyone is playing weird, you just simply can't control risk (even if you are making logical plays).

I'll admit that I had a few of the equities that I held before the crash over the summer drop by 50% percent. There were people out there talking about being glad that they got out and them going on about how much better it was for them to get out early. Some of those tragically massacred equities I held came back to nearly their original prices and I somewhat regretted not putting some more cash into the market. I can't expect to trade with 20/20 foresight and I'll take the bad with the good.

Sure I am a bit down from the market haywire but I am happy to bide my time, earn money from work and wait. More importantly, instead of waiting, have other things to do :)

Went Jogging for the Halloween Weekend

I've been out of Tokyo for the last few month as a result of work. I've probably fallen off the face of most of my social networks during that time and at the same time, I kind of enjoy the peace and quiet from people. I used to be really busy every weekend meeting up with people somewhere, attending events. But it is nice to spend a weekend home.

I woke up, spent the morning doing some research, went out for a jog, came back, had a steak dinner and had the best sleep ever. I naturally woke up at 8 am today feeling particularly great. This is something that I need to repeat more often.

Spend time, doing productive things, go to bed tired from hard work and wake up great. This is how life ought to be lived.

Deploying to AWS doesn't make sense (not just yet)

I've been mulling the thought of deploying a webserver to Amazon's AWS service as of recently and I've been running the idea through my head multiple times and something just didn't sit right. I've been playing around with a webhosting service for the past year to see what it is like to work with a webhosting service and came to the conclusion that the lack of control is a killer data portability; it is easy to create a blog and write on it, but it is a real pain in the rear to back it up and be able to move it elsewhere should I want to. The other problem is the limited control of software installation I can do, meaning that I would be completely limited to whatever the service provider has installed. This is where Amazon's AWS service comes in.

Amazon's AWS is basically a server instance that you can use, it is very much like having your own computer to play with, install an operating system and then do whatever you want with. I've been doing calculations on price for such a service and the numbers don't look favorable for a single "Small PC" instance which runs for about $744/year. At that price, I could buy a stripped down computer for far less and turn that into a usable server for at least a few years. Transitioning to AWS only would make sense if I needed signifcant computing power beyone what I would provide from home. 

I'll be looking into buying a new computer to play with at home.

Rusty Debating Skills

I lived in Sendai for the first 3 years of my life in Japan and one thing about living in Sendai is that people don't generally debate and disagreements are often avoided. Moving to Tokyo was rather different as being good at making arguments is important, especially when working in a company.

I, unfortunately, have not developed my Japanese debating skills and for the lack time spend among English speaking peers in Japan, I have found that my debating skills gone rusty and the "thick skin" that one needs to survive a debate has also gone thin. In effect, I believe that I've lost some mental agility and stopped keeping a pool of readily accessible information in my head to make arguments because I've had no need to before. Information retention is terrible at the moment, though some might attribute this to the aging effect, I refuse to accept that as an excuse.

Especially when it comes to dealing with upper management, when making persuasive arguments is especially important for these people to sway executives and push their career upwards. If you want to do well in a career, having good debating/persuasive skills (balanced with it not being aggressive) is absolutely important.

To be honest, I miss my old philosophy class in high school where we used to debate on a variety of topics at 7 am. Sure we would be sleepy as hell, but it was a lot of fun back then.

Playing with some new styles

I've been using the old blog style for ages and figured that it was time for an update. I have no clue when it comes to website design so I just decided to play around with whatever they had on blogger. Not too excited about the new look because I think the templates they have here look too goofy. I'll just suck it up since I can't go back to the old design now and I think that most people that read blogs anyways do so through google reader (and if you don't, you should!)

Kick ass and then become a consultant?

Into the second week of October and already cranked  out more than 20+ hours of over time. If this was Canadian style flex time, I'd have 3 vacation days stocked up. But this is Japan, so I'll be taking overtime pay instead.

Been working on putting together some software for a project for wafer level analysis. I use a newer language compared to the older guy I work with, Visual Basic vs Python. Programming lightening fast while I have most of the base engine of my detection engine up.

The problem here is that by the end of this year, my stint in Kagoshima will probably be over and my work will need to be passed off to the guys here to continue. The problem is that no one can program in the language that I use. They'll have one of the guys eventually translate the algorithms into their native language.

Interestingly, I really do wonder how often do cases such as these occur, where one guy is able to do a good job that the other guys that follow after can't keep up. I best that this is how a lot of people start their contracting gigs.

Steve Jobs has Passed

In an instant, this is the only news that is making the rounds on the internet. The iconic Steve Jobs is gone and there is a void in the world of technology.

Some links to postings about Steve Jobs from notable people:

• Bill Gates
• Larry Page 
• Sergey Brin
• President Obama

Outpouring from the internet is simply massive right now.

Is a crash on the way?

I know for sure that a huge rally is probably not on the schedule, but there ought to be either a crash or stagnation in the stock market. So long as you have the timing right, selling calls during times of high volatility would be smart to me. I don't hold any short positions yet (I wished I did) just because I haven't had time to do sufficient time to get myself into the right positions.

Of an interesting note, the venture business in the US could be looking to take a hit in the slowing economy and this is something that I'd like to keep an eye on.

As for now, the yen is sky rocketing like crazy. What that means to me is that I'll be looking in ways of putting in more over time to make more cash and sell yen for other currencies because I think the yen is over values.

From Kagoshima to Tokyo and back again

Finally got back to the home after 2 months in Kagoshima. Getting off the plane and getting into the city, I realized that I was facing some culture shock from being back. All the people and tall buildings... the other thing I missed while being back is good country food.

I eventually made it back to my apartment and found that I had left a light bulb on for the entire 2 months. An interesting experience but when I did power calculation, having that light bulb on for 24/7 over a month it cost about $24. It's pretty amazing how much power a light bulb draws over the course of a month, more than the fridge I have.

But to be honest, most people don't have their lights running at home for 24/7. Interesting lesson.

The mail box at home was stuffed to the brim with fliers for all sorts of things, especially girly stuff like manicures, hair salons and frilly clothing- nothing that a man would need. Sexist marketing at it's finest? Quite possibly so.

I also dropped by the office today and saw one of my old managers and his greeting to me was:

Manager: "Welcome back. You look thinner. You okay? Maybe you should see a doctor"

Apparently I have a rather interesting work place.

I got back and picked up a $800 LCD monitor and had it shipped off to Kagoshima for me to use for programming. I even had it shipped by air. One of those perks about working in a large corporation.

Applied Stock Market Stochastics 2 (applications to options)

I wrote earlier in Applied Stock Market Stochastics about learning using the mean and the standard deviation of stock market data to calculate the probability distributions of where a stock might go in the future. I've decided to take what I learned from the previous post and take it a step further in application to Option derivatives.

Understanding options is a little math heavy; option pricing comes from the black-scholes equation pricing equation by assuming that the price of an equity moves in a Geometric Brownian motion like way with a constant drift and volatility. We can model this using the mean and standard deviation by throwing the parameters into a Gaussian function. The black-scholes equation is basically a closed form solution to the differential equation representing this condition.

I got tired of simply just looking at the equation so I decided that it would be more interesting to create a statistical model to determine the motion of the price of a stock. In my last post in Applied Stock Market Stochastics, I analyzed the historical prices of the Dow Jones index as a distribution of percentage change on a month by month basis. The result showed that the motion of the stock market moves around in a Gaussian like manner.

I will assume that you have a basic understanding of put and call options. If you don't know about them, the short story is that the purchase of one of these contracts is a bet that the price of some equity will be above or below some price. If it is above or below that price, your profit is proportional to that difference (and if you need to learn more, hit the links or Google for more details).

With that in mind, the question that I want to solve is the likelyhood of making a profit or loss and by how much. If we understand the probabilities of making a profit or loss we can then calculate for the "expected value" of the trade-- which is the average loss or profit that we expect to get if we made the same bet over and over again. If the expected value is positive, we expect to make money and vice versa.

Now, I am not aiming for the level of sophisticated analysis hedge funds do out there and I won't pretend that I know that much either, but I do have a great fascination of learning about things though a first principles approach; and I think this is a great first principles approach to tackling this problem!

So let's first model the motion of an equity instead of a stock market index. In this case, I've downloaded Google's historical prices for the last 3 months (between June and September) and calculated the mean and standard deviation of it's % chance in price on a daily basis. The numbers I got from this calculation were 0.0373% and 2.409 for the mean and standard deviation in percents. For actual use in the models, I divided these numbers by 100 and ran a simulation using 100,000 simulations to simulate the resulting price of the Google stock by October 20, 2011 which is the expiry date for a set of call and put options. The resulting plot looks like the one below:

The change factor is the multiplier which you would multiply the stock price to determine it's final value. The plot here, is also normalized (meaning calculating the area of this graph will equal 1 or 100%).

Now for illustrative purposes, I took an option at random, which was the $555 strike call option dated for Oct 20th 2011, which was priced at about $12.00 when I checked at Google Finance and plotted out the returns as a function of price:

$555 strike, call option Profit/Loss profile @ a price of $12.00

So now we have now have 2 pieces of information: the probability distribution of Google's stock changing by some factor by the option expiry date and the Profit/Loss profile of the stock option. Now the most important question to answer is, given the current price of Google's stock at $537 (when I checked today), what is the weighted profit/loss as a function of price? The answer to that is to take the probability distribution of final prices and multiply it by the return distribution of the option to get the following plot:

What this graph tells us is what you expect to earn/lose over all the possible outcomes. Summing the values together will yield the expected value of this trade, or the amount of money you expect to make when making this bet over and over again. I've done that calculation and the number resulted in -$3.023/per share (and 1 contract is a multiple of 100 shares) meaning that you expect to lose $302.3 on average when making this trade.

But!! There is an upside to this however, in that you can instead sell a call contract instead of buying one and expect to make $302.3 instead and turning this into a profit.

Caveats

Obviously, if this were the case then everyone would already be doing this to make money in the stock market (and to some extent, there are people and organizations out there that are already doing this). So there are 2 important assumptions that you needs to be aware of, and those are:

1. that the motion of price can be modeled by a random Gaussian variable and
2. the mean and standard deviations are accurate and do not change

But there will be times when Google's stock price won't behave in the random manner that we modeled it as and we don't event know that numbers that were put into the model were right in the first place! This is the risk we take when using this calculation (and any other model for weighted profits and losses!).

So what is this good for?

This is a problem that plagues historical analysis when trying to make predictions of the future. But the most important take away from this is developing an understanding of the likely and unlikely future outcomes to help in making better decisions; in the long run, this is the most important part.

Now, even with these points of uncertainty, this could be used as a model outcomes based on your research and how you feel about it. The great power bestowed from this is being able to quantify your opinions of the future and risks involved by turning them into an expected value. This helps considerably in decision making instead of simply just going off on "a hunch."

There is obviously more work in modeling random variables and outcomes but I hope that you find this a useful tool in thinking about the future movement of a stock. Ideas and comments are most welcome.

Load Speeds are Important

I bought some hosting space on iPage in February of 2011, the service was cheap, space was unlimited and I guess I got what I paid for when it came to speed; fairly slow load times. I wasn't happy enough with the load times to move the contents of my blog over to the site and domain that I've been playing with. But I'll make the appropriate announcement when I finally find a decent hosting solution.

I have been toying with getting onto Amazon AWS and setting up my own server and having my own space. The great thing is that I would have some persistent computing power available to me to run some automated applications. I really think that there are many fun things that one can do with having a server, without having the need to own the hardware to do things.

I am somewhat reluctant at using my main computing box at home as a server since I want something that I could completely crash with impunity but I am not really that interested in having a second full blown desktop computer at home. One option that I've been contemplating is using a MacMini as a server due to its really small foot print, the price is reasonable too at about $700 for a system with 4GB of memory.

Interestingly the same system sells for about $100 more when buying through the Japanese Mac Store. I think the exchange rate is playing with the price significantly as the Yen has sky rocketed (time for the arbitration??).

Time to do some research, but with some effort, it is possible to setup a server on AWS for about $110/year and have some fairly decent speeds. Storage goes for about $0.10/month meaning that I could have 4-5 Gigs of website data easily and still be only paying about $0.50/month in storage. Seems really reasonable to me.

All I would need to test is latency and bandwidth. If things look good, I'll probably make the switch.

Distractions are Productivity Killers

My work setup currently has me working on a large rectangular open desk suited to 2xN people, where N is determined by the length of the desk. The office consists of a large room with M tables where everyone works at. The problem with this setup is that the office can get really noisy and I have a hell of a time concentrating when there are that many people around, all conversing about a wide range of different topics and phone going off every 15~20 minutes.

In an environment like that, memory and concentration intensive work is nearly impossible. To be honest, some of the best progress I've made with code was done while working the night shifts where no one was around and I was able to just zone out and simply focus on what was going on with the code. While programming, I really need to be able to keep track of variable names, the structure of the program and algorithms in my head. The instant my concentration is broken, it takes time to reorient myself to figure out what I was doing again. This is actually a classic coding problem, especially when looking at code that one wrote days earlier.

To me I am not a huge fan of large open offices. There are some people out there that are able to focus in such environments, however, I am just not one of those people. I'd much rather have headphones on and be dead to the world while getting work done.

I am quite sure that offices around the world are setup in some manner like this, but I think it is rather counter productive. With laptops being used a primary computing devices, I really don't mind having a main desk somewhere, but I'd be much happier to have access to a silent stall where I can allocate blocks of 2-3 hours to get some serious work done and return to my desk to get less mentally intensive work done and be accessible to people. I am sure that there has been plenty of research done in this field and I am curious at how much of this research is implemented in offices around the world.

I would venture to bet that an architect that is able to create highly productive environments would be paid big bucks!

All You Need is the Right Trigger to End Microsoft

I started using Ubuntu Linux about 3 years ago and currently going on to my 4th year using the operating system. There were some hiccups at first when switching over, but now looking back, I have found the process to been a worth while one.

The greatest benefits to me about using a Linux system is the ease of reinstalling software and getting original settings back after a fresh install. I still remember times when working on a Windows computer where I would reinstall the OS once or twice a year to freshen things up and spend 2~3 days getting the system back up again with all the software that I use and configuring everything to my liking. What a pain that was.

The difference with working on Linux and having a home directory where all one's settings are stored in (somewhat) hidden folders, is that as soon as program is reinstalled, the home folder is the first place the program looks for to get the setting information. So long as you backed up the home directory, the instant you reinstall software, it is automatically configured. The other great thing is that software can be batch installed from repositories, and instead of having to deal with installation dialogues, you just tell the computer to install the software and it automatically installs them to a binaries folder and things are automatically setup. If you have multiple users, everyone's home directory holds each user's personal settings so there is no clashing whatsoever. Everything is simple to install and fast to restore.

The other great thing is having free software and having a repository to download whatever you need from. Just download the software, give it a try and if it works great then keep it. If not then uninstall it and try something else and most free software out there is "good enough" for all practical purposes.

There have been times that using free software didn't work out for me because of some minor formatting or incompatibility issues. For example, while open office can open and read MS Office files, there are times when the formatting of documents get a little off. I had the unfortunate problem of trying to write and update resumes in open office to find that the alignment of text was just off enough to make the document look a little unprofessional. As a result of that, I've become a little bit of a fan of PDFs, once you convert the document into a PDF, it should display the same across all systems.

If worse comes to worse, with the advent of OS virtualization, I am now able to run Windows inside of Linux or Linux inside Windows or what have you. I think this is a great boon to trying out new OSes and also getting access to just that one application that you can't find on your native OS. Rebooting is such a pain as I generally don't like needing to close all my applications and then open them all again after restarting the computer. Right now I use Win XP in side of Ubuntu just for MS Office on the rare occasions to edit documents, but other then that I don't use Windows for much else while at home.

To be honest, I don't see any real argument for the need to be dependent on a single operating system anymore. Most of the best software are multi-platform now and using them is just a matter of installing them onto whatever OS that you're on. Even writing multi-platform applications is not terribly hard using languages like Java or even Python since just-in-time compiling is sufficient to create native code for the applications to run.

So there really is no real need to be totally dependent on a single operating system anymore for any reason other than for the comfort of a familiar interface. To be honest, I think that Microsoft is walking on hollow ground; the instant the ground gives, I think it will be all over for them.

Is Facebook killing blogs?

One thing I've noticed recently is that I am far more spontaneous when posting pictures and updates to Facebook as opposed to writing on the blog. Uploading pictures to Facebook is dirt easy, albums are automatically created and linked to my profile and it is quite easy to make short posts while on the go. That and compared to the blogging medium, one may have to upload pictures to a separate site, then add comments and then link back to the blog. It's the multi-step work that is a certain killjoy to me.

I've tried blogging while on the go (on places like the train) but found that doing things like looking up multiple webpages, adding links and doing basic editing (like moving a few paragraphs around) to be quite cumbersome compared to writing short status updates.

Which brings me to the point that I believe that traditional blogs are dying out compared to posing things on the fly as smart phones become the norm. The vast majority of my friends have since stopped updating their blogs and mainly post news only on Facebook, which has been the trend for quite some time.

What will happen to blogs in the future, I am not entirely sure. Perhaps one day there might be a big enough backlash against Facebook that people may return to realm of blogging. Who really knows. I for one will continue to write, and actually make a point of spending more time posting to the blog compared to writing on Facebook. Perhaps the next best thing is stronger integration into (out of?) Facebook through blogs.

To be honest, there isn't a whole lot of original content on Facebook anyways and much of my best thinking happens while doing "real" writing.

Applied Stock maket Stochastics

A stochastic variable is pretty much a non-deterministic variable, in other words a random variable. I've been thinking about stock market in terms of statistics to better quantify risks and expected values because it really boils down to understanding the risk-reward profile and creating a portfolio with the desired characteristics given the available financial vehicles.

The challenge is then quantifying the risks and rewards of which is a black art. I doubt that anyone has a perfect system of setting these values a priori because risk and reward are particularly fuzzy numbers until after the fact.

Given the current fragility of the world markets, one occurring theme that I've been contemplating is the mechanics of hedging. There are of course several ways of not losing money before a market downturn may include:

1. Selling off equities
2. Buying bonds
3. Shorting stocks, selling calls or buying puts
4. Buying short ETFs

The list is not extensive but those are some of the options that one would have during market uncertainty. The next question becomes, what do you think will happen in the next time frame (which can include spans of days, months or years). The idea will be to track current and future issues and setup a portfolio that tracks your sentiment based on the data collected and your interpretation of them and make bets on sentiments that you feel the most confident about.

Suppose that you don't believe that the stock market is quite volatile but not in a position to make huge gains. If you have securities already and already made significant gains but don't want to incur capital gains taxes, selling your securities might not be the best course of action. In this case it might be wise to lock in gains by hedging against future drops.

This begs 2 questions:

1. What is the probability of the stock dropping n%?
2. Give the probability of a drop, how much should you pay for insurance?

Now obviously without part 1 of the answer, answering part 2 is going to be hard. So I've started first looking at part 1 of the problem. What is the probability of the stock market moving in some direction in some time frame?

The most obvious method that came to mind was to first get the probability distribution function of a stock market for the range of all monthly percent changes (I could have done daily, but I already crunched the monthly data), The graph looks like the one below:

 

The above plot contains all monthly data from 1928. The plot looks fairly Gaussian yielding a mean of 0.764% and a standard deviation of 4.445.

The next step that one takes with this data is determining where things may go in the next few months. The easiest method to calculating what may happen in the next 6 months is to create a random Gaussian variable with the mean and variance of the Dow Jones index and multiply them together to determine the probability of what may happen in the next 6 months. The result is the following graph:

Given the cdf of the next 6 month, the next question to answer is what is the probability of seeing the Dow increasing or decreasing by some factor? This requires some integration but I've done some sample calculations and those results are provided below:

Based on the information below, we have a general idea on the probability of seeing growth in the Dow based on all historical data within the next 6 months.  What would be more important is to find means and standard deviations that represents the market right now; which could be done through calculating the distributions of %changes in price over a more recent time fame, or more importantly, values that they might expect in some future time frame based on current research.

The idea will be to create a portfolio that represents a personal bias on individual stocks and the stock market in general to create an adequately hedged portfolio against catastrophic crashes while exposing one's to certain amount of gains to mitigate risks while trying to aim for the most optimal profits. Modeling the motions of the stock market in general are important tools to help create a portfolio that matches one's risk-reward profile.

This is still a work in progress with many imperfections but it is a start to creating portfolios with explicit characteristics.

Why you shouldn't try to become the perfect trader

I've been going through historical data in the Dow Jones recently to try and better understand past trends. I've mentioned before that I generally try to stay away from technical analysis but I've decided to put away that hat and decided to see where this takes me. The one good thing that looking at historical data is understanding the scale for change, things are measured in months and years instead of days and hours. I have a terrible habit of looking at daily market data, thinking that motions in the day are indicative of where things are going to be in the long term. So I'm going to stop looking at the market like that (though it can be fun sometimes... sort of my equivalent of "watching sports").

I've simply started plotting through the Dow Jones looking at annual returns year by year. Looking at annual returns is a really arbitrary thing to do since I am picking a price at some "random" date and comparing the ratio of it to some other date in the future. I am sure that picking different windows will have different values but what else am I to do, eh? Not a whole lot until I figure something out, but for now this is it:

I've grabbed annual data back till 1971 and the funny thing about looking at 40 years or annual data is that there is so little data to look at. The first thing that pops out at me is the all negative change years occur after a +10% change year (though not after all 10% years). The second thing is that 5 out of 7 negative years turn positive the year after. So what would you do in this case? One could surmise that it might be a good idea to take out some money after a +10% year and put it elsewhere as a contingent against a negative year (Ie. Bonds or something). The next thing is that you might once a negative year occurs, you have a good probability of catching a positive year after a negative year. Something like that could work but the problem is that you will still need to optimize against having 2 consecutive bad years. There is also a period in the 1990 where there was nearly no negative growth years, but let's say that you've optimized your strategy to factor that in too.

Now suppose that based this 40 years of data you've made a fairly optimized portfolio where you take a small chunk of money during good years and put them into bonds when you make more than 10% when the new year hits and made a rule that if there is a negative year, that you would plunk (say) about 60% of the money you had in bonds into the market during a bad year to catch the probability of a good year the year after and it left you with enough funds to deal with 2 consecutive bad years you could leverage yourself since the probability of having 3 bad years according to this data is zero (the alarm bells should already be going off here).

Suppose that this strategy worked well against this data, you might be fairly tapped out if there are 2 consecutive bad years and the strategy provides for very good gains. Unfortunately, if you look far back enough in history, there is a period right after 1929 where there was 4 consecutive bad years. If a black-swan event like that happened, everything in your portfolio would be gone. Unlike poker, you don't generally have "rebuys" in the stock market (unless you have really rich and possibly foolish relatives), If you're going to want to play this for the long game, developing a methodology to understand what is going on around you, knowing your options, knowing your limits and knowing what exactly you are willing to risk is important. Having good well founded rules to maximize your success are important and you should, in general, play by self set rules that are designed to maximize your chance of success (and though, just like poker, success isn't guaranteed). And just as a note, there are 4 years of consecutive negative growth around 1929 that might have wiped this strategy out (then again, it probably wiped out nearly everyone during that time). In addition, you may want to have a contingency to catastrophic events.

I had this knowledge internalized in me in poker so well that I had forgotten to be aware of my general rules that made me a generally strong player. Most of my poker playing rules were also mathematically generated too, the game of poker (un)fortunately is really easy to analyze. All you need is some general understanding of probability, expected value calculations and knowledge of permutations and you pretty much had it made. The stock market is different and choices and information is so much more vast such that it nearly impossible to understand the entire system... though I still do try (though I am off on a tangent).

Rule number 1, though, it to have a set of rules to play by and more importantly, have a set of mental interlocks to prevent yourself from going "tilt." Getting into the stock market will make one incredibly emotional. A bad stock market period is like PMS for men (that is not to say that only men participate in the market) and you honestly need to get a hold of yourself during those time because you are going to be prone to doing stupid things.

This coming of the second crash (?) has given me a great opportunity to better look in to and handle crashes. The first stock market crash that I have ever had to have deal was the 2008 one, and my, that was a doosie. I wished I was old enough to have participated in the 2000 dot com bust, because I would have been much younger and had a lot less money to lose. Unfortunately, I was too young and had very little real capital to my name during that time (and the cost of trading was way too expensive during that time; $29 for a single trade!). Anyways if/when I do have kids, I am giving them money to blow in the stock market, they might as well learn to lose everything early and fast (and I hope to not make chronic gambling kids while at it too).

Fail Safe (avoiding gambler's ruin)

Unlike other games, the stock market is the kind of game where you play on one "life." The idea that you're supposed to be in the market till about 65 ish (or longer) and not screw up and lose nearly one thing is actually daunting. It's like pretty much saying that for 30-40 years that you are not allowed to fuck up. But that is pretty much it, lose once in the middle of it and you are possibly screwed.

I've come to the realization that optimizing for gain and building a robust portfolio are (to me at least) opposing factors. Optimization is basically trying to allocate resources perfectly to come out with the most perfect result as possible (which is easy in games, hard in real life) because it can equate to doing things with minimal margin for error and going for possibly unrealistic perfect results. When it comes to investing, you are never going to get the most of that up swing, nor are you going to know when the market has hit bottom. The question is not a matter of making the perfect play (because it is impossible), can you make decisions such that are "good enough"? The next important thing is that can you make consistently make "good enough" decisions? Because if you can, then you are on your way to becoming a good investor and I think this is often an overlooked point.

The important thing about investing is being able to quantify the risks of every decision that you make and have a general idea of what you are going to do if things turn bad. How much loss are you willing to take? if losses do occur, are there other assets that can balance against that loss? Is the portfolio setup in a way that you won't be psychologically affected if things go to bad (or good?). Risk quantification in a portfolio is really tough and it is a really fuzzy science but it is very necessary. One of those lessons learned from my crazy crash and burn in 2008.

How am I faring?

I think I was lucky this time because I wasn't overly invested in equities and though the equities that I did hold lost a lot of value, I am sleeping better compared to last time. The other thing I should note was that I was a little impatient in getting back into equities a few months ago when it seemed to me that things were going to start taking off. The equities that I did end up buying were over priced because I thought things were going to be a little better than they currently are right now. I should have been more patient but that is the way how hindsight works.

I've also thankfully kept a good amount of cash out of the markets and will be patient for things to cool down in the market before looking at getting back in. The volatility will be here for a few months until Europe sorts it self out and people get over the S&P downgrade of the US (which ironically, did something good for US treasury sales). There will be plenty of sucker rallies and swings coming up... but then again, that is already pretty normal for the stock market.

If you can make sense of the following data, then you deserve to be a millionaire.

A look at some past crashes

I had some spare time over the last few days so I've been going through some of the Dow Jones historical data with some programs to study volatility during some previous stock market crashes. I first plotted out daily percent changes in price and volume to look for large swings in price and volume. The result of that plot is provided below:
Looking at the percent change in stock price, it become fairly easy to pick out periods of time of high volatility and we see a few points in 1987, 1989,1997,1999-2003,2008-present. By the looks of the graphs, it takes a few months until volatility subsides. Though as of today, it seems that the stock markets are up, I highly doubt that the choppiness seen in the Dow last week will be the last.

The % change in volume plot doesn't capture well changes in volume since it could simply mean that volume could have suddenly risen and remained constantly high during a crashing period. Though from the plot, we do see that volume can vary quite wildly.

Given the % change in Price plot for the Dow Jones, I had a program go through the data looking for points in time where a > 5% decrease in the index occurred and extracted 4 months of data before the crash and 6 months after the crash to get about 10 months of data. I also had the program mark points where a > 5% decrease occurred and came out with 7 plots:

Black Monday Crash

Friday the 13th Mini-Crash
Asian Financial Crisis

Russian Financial Crisis

Dot-Com Bubble

Sub-prime loan Crash

US Debt Down grade Crash

The data seems to indicate that during the pre-2000 eras of crashes, volatility was far lower compared to what we are seeing in the markets now. This is probably the result of the HFT algorithms going at it in the market place. The most recent crash is showing far more volatility compared to all but the 2008 stock market crash as we have seen in a single week, some very large price movements in the Dow Jones Index. There was also a large spike in trading about a week before volatility picked up and I am not sure what this represents.

Not really sure where the volatility is headed from here, but there could be more on the way if things this time around play out in a similar way to 2008. We'll have to wait and see I guess.

Where does "net worth" go during a stock market crash?

Understanding your "opponent's" trading account is akin to keeping track of the number of chips your opponents have during a game of poker-- the amount of cash your opponents have will have an impact on the way they play the game.

One thing about stock market crashes that fascinated me is trying to understand where the money and capitalization value goes. I have been working over the idea of the existence of "phantom capitalization" in the stock market as a result of stock prices being bid up and I was interested in trying to understand the mechanics of a stock market bubble. So I was basically trying to model a feedback loop were people start pouring more and more money (and leverage) into the stock market, thus creating the illusion that everyone has more equity via "phantom equity." The process repeats until it is unsustainable and things fall apart.

I believe that there is a disconnect with this mode of thought since people are fooled into thinking that this equity is real and the harsh reality sinks in when the inflated stock prices tumble which wipes people out. The real objective of participating in the stock market is to buy and sell stocks to eventually cash out with more money than you started. By playing by any other rules instead, I would argue that the investor is fooling themselves when participating in the stock market. Unfortunately, the calculation methods used to value equity is somewhat faulty and is prone to enticing people into inflating stock market bubbles.

The first thought that came to mind was to try and simulate players and their interactions in the stock market; trying to understand the impact of their buying and selling of equity and how it would look like in their trading accounts. Obviously, this is a pretty hard problem as the real stock market has many individuals and tradable securities. Going for the full monty is obviously too complicated so I decided to sandbox the whole issue and have 2 players with cash and a single tradable stock which both players owned at the beginning. When one player issued a buy order, the other had to sell at the same price and volume and by using this simple model, I wanted to see what a stock bubble would look like under this isolated situation.

I managed concoct a trading pattern between 2 players where they could buy and sell and equity at increasing prices while keeping the amount of money and the number of stocks they had on hand constant. However, through the act of trading and using the standard method of calculating the "average action price" of each person, I was able to inflate the price of the stock and thus was able to create phantom equity for each individual. I have included a screen shot of the simulation I performed in the image below (click for the full sized image).

In this rather simple simulation where I kept the volume constant, I noticed that the swings in the "cash" column got larger and larger as the price the stock increased. In more realistic situations, I would presume that the volume would taper downwards to reduce such large swings in one's capital account. I would surmise that this also wouldn't apply to just individual investor but is also true for computer systems engaged in high frequency trading. But the main point is that it is possible to observe the inflation of networth of 2 parties trading in a single market and that it should be possible to extend this model to a many people market with many different stocks.

For the above calculation, I didn't take it all the way to the point where both persons were leveraged to continue bidding up the equity price, but it should be quite evident that the swings would get larger until the point where neither person A or B would be able to leverage themselves any further. Supposing that a dormant person C existed, any sell off by this person would result in a catastrophic crash as margin calls are made.

From a technical trading standpoint, I think it would be an interesting exercise to find ways of watching the volume and price as an indicator of bubbles in the stock market. One concept would be to watch for decreasing volume while prices rise as an indicator of a bubble since higher prices will equate to lower volumes as a stock is traded up. I would also expect that the converse be true in that volume would likely increase when stock prices are low since it takes more of the same stock to move the same amount of money.

What I am very interested in trying to figure out as a result of this simple calculation is an empirical method of guessing at the "action prices" of other participants in the stock market. I have a hunch that people will have a tendency to sell off their stocks after turning a profit and will likely end up spending their profits on buying other (inflated) stocks. The whole process continues until a bubble occurs and it pops, leading to the destruction of phantom equity.

One method of making money in the stock market is simply knowing when to get out and staying out when a bubble occurs. And the trick to understanding when a bubble is occurring is figuring out when there is too much phantom equity vs the amount hard cash available.

The Unwinding yen carry trade?

The yen is skyrocketing recently against the USD, CAD and the AUD over the last few days. The chart for the JPY vs the AUD is especially astounding as people are unwinding their carry trades. The recent spike is astounding. The other thing, however is that I am quite certain that the rising valuation of the yen is unsustainable against other currencies and there should be a correction happening eventually. The question is at what price and how volitile things will be during this time period.

One thing that I appreciate after coming back from Cambodia and Vietnam is an understanding of the bartering system, where the prices of goods is often bartered between the customer and the store clerk. Prices do vary between shops and it is very interesting to see how prices can be very fluid between places. Everything is driven by the sentiment of the store owner, the customers and other market circumstances and this experience is equally applicable to the stockmarket.

One intersting observation that I made about the merchants that I bartered is that they liked to ask where I was from. Basically, what they were trying to discern is what price would I be willing to buy something by the living standard of the country that I lived in. A Chinese person might price things differently from an Europeian and etc. When it comes to the stock market, instead filtering the buying and selling patterns of people by country, one would need to understand the trading motivations of different financial institutions. Meaning that different people would have different ideas of what a fair price for the same financial instrument.

Anyways, I am very tempted to sell more of what yen I have into other currencies, but the question I am trying to figure out is how long will the sell off in the stock market will continue. I believe that the yen will continue to strengthen the worse things get in the stock market. The question is when will the sell off stop, at what price will people start selling yen and etc. All of this is really hard to figure out.

When it comes to the financial market, one cannot look at companies in isolation but also other traders and their motivations in the market. I have a new appreciation for that and very interested in developing tools in trying to understand the financial conditions of the other players in the market. As in poker, you have to be able to keep track of the number of chips everyone else has because how people play is also dependent on the number of chips they have.

In Cambodia

This is probably out of the blue, but I am in Cambodia right now. My work had a scheduled 1 week shut down of the office and facilities as a result of power conservation efforts in Tokyo. Apparently, companies have a sort of power usage limit between now and until mid-September and going over the power limit will result in fines based on the number of hours a company is over the limit. There are other details but I won't bother getting into them as it will probably detract from the main contents of this post, as in me being in Cambodia.

Dispite backpacking through South-East Asia 3 years ago, Cambodia is one country that was on my list to goto, but I never made it to because of time constraints. This time around, with the 1 week holiday from work I would go and hit Hanoi and Siem Reap to see the Angkor Wat temples. After a bit of a turbulent jounrey from Vietnam to Cambodia (not to mention that they also block facebook there), here I am in Cambodia and I have some pretty positive things to say about the country.

The people are incredibly warm and after getting out of the airport there weren't a bunch of taxis waiting outside waiting to over charge travelers to get into the city. We also had a tuk-tuk (a motorbike connected to a sort of trolley) that waited patiently even as my flight into the country was delayed by 2 hours. Compared to other SE-asian countries, there is a lot less price gouging here; then again, the cost of living here compared to other countries is so much cheaper that even if they raised prices, I might not even notice.

The price for a dish or something from a local stall ranges in the $1-$2 range, a foot massage for 30 mins is $2.5 and a full body massage for 30 mins is $3, just to put things in to perspective. Staying at really nice hotel runs for about $30/day, which is really reasonable. My imagination runs wild with the idea of running off here and not returning to work for simply years while staying out here.

The other nice thing about people here is that they are generally good natured and even if they can't get a sale out of you, they still are nice to you. Which is a rather nice touch. One of the guides I met is only but a teenager but has managed to learn a really good level of Japanese through 1.5 year of study and acts as a guide for many Japanese tourists that come out here. The determination I've seen in some of the youths here is simply astounding.

There is one sad thing that I've felt while being here is that there is a huge influx of tourists in Siem Reap and it feels like that many of them are just here for the cheap thrills of a country with a really low cost of living- there are some people out here that I've see that I would have a hard time imagining getting anything of value from visiting the angkor wat temples and at the same time, it feels a little sad that the locals are doing their best to cater to these people through the creation of bars and restaurants to turn a profit-- a perfectly good environment being ruined by the influx of money through the western world... and that was probably the most profound insight I had while walking around the city today. I took a bunch of photos, but haven't had the chance to download and sort them yet. Hopefully, I'll have some time to do so later on.

Being shipped off to Kyushu for a few months

The current phase of the project that I am working on has ended and as a result of that, it seems that my expertise are required elsewhere in the company. Crunch time is coming up for a special Sept deadline and I was just called in by the boss, yesterday to learn that I am being shipped off to Kagoshima, Kyushyu for 2 months to help troubleshoot some process related issues.

They'll be putting me up in a hotel for the first week and then a mid-term housing arrangement while I am there. I'll likely be working a few graveyard shifts as well... the compensation for taking this job is still being hammered out but here is hoping that I get some good spending money out of this.

Working on a few other things on the side as of the moment and the weekends have been really busy with events.

In other news, I'll be taking a 1 week vacation in the first week of Aug to visit Vietnam and Cambodia. Posting will remain light...

We are in for an interesting FX ride

Been following the news a little bit lately and it seems that Greece and Italy are having debt issues and the currency markets are going nuts.

The US will need to pass a bill to raise their debt ceiling on Aug 2 and what exactly will happen there is uncertain but I am quite sure that they will raise the debt ceiling, because it will have significant impacts on their ability to raise money in the future if they go into a mini-default.

As a result, I've just been looking at the currency charts recently and the yen is jumping upwards in the midst of this uncertainty. I've already diversified some of my funds out of yen an may look to purchase more foreign currency as I think that the markets will be fairly unstable.

It will be interesting to see how things will play out over the next few weeks, but I'll be lining up a cascading orders in CAD and AUD should the yen spikes up some more.

What does it take to run an E-business

I've been mulling the idea of having some parts fabricated and sold online recently and got down to considering the challenges of logistics and putting together a small site to see what happens. It's more of an experiment than anything else but having sold some X-box's though E-bay before, I just thought: "how hard would it be to create an online store and be able to accept credit card payments for something."

Been looking around here and there but a cursory search indicates that one can open a merchant account for about $25/month through paypal and deal with them skimming off some of sales. Modeling selling things online is also quite fascinating and what I noticed is that sensitivity to profits is very high when margins are low. The most important thing is learning how to create a landing page and figure out how to drive traffic to a site to gague interest before starting.

Premature Optimization is Bad

The title almost sounds dirty, but still, one shouldn't get too excited and ahead of them self when it comes to implementing programming solutions. One of the biggest challenges to programming is understanding the full scope of the problem that is dealing with. When I mean "problem," I don't mean it in a sense that something is broken, but in the sense of understanding the "specification of the system" of what one intends to build.

I am currently putting together a more advanced parser that can read text and extract values out of it from html. With the advent of content management systems (CMS), most of the data displayed on the internet follows a specific layout depending on the page. Writing parsers can be a tedious process as you need to do several things:

1.    Effectively isolate the block of text want to analyze
2.    Write a parser to target and extract the information you want to get, and
3.    Create a data structure to save that data

This process is time consuming because I would have to write additional code to parse different values and change the data structure to save everything. For every additional dependency there is in code, the probably for error goes up exponentially (a gut feeling). The less dependencies between modules there are, the less errors in code that you'll probably get. The best kind of code is code that automatically adapts itself to whatever you're doing... but I am going off on a tangent. Right... dealing with premature code optimization.

I have come up with an idea using html templates instead of writing code to parse data out of an html block. Writing procedural code requires one to think of a strategy to get at the data wanted and then codify the process in a program. That means that for every block of html, one would need to write code to get at the data. I've done that before and it can be a time consuming process (which can be made somewhat easier with the use of parsing libraries however).

Using templates it becomes pretty easy to specify the structure of the text and target certain sections of the data using keyword markers. Editing text and converting them into templates can be somewhat tedious but still far easier than writing code to extract data. The work required is just turning parts of the template into wild cards and entering keywords to certain parts to get at the data and I really like this solution, so much that I decided to write code that would semi-automatically take html text and convert it into a template. It took me over a day to try and put something together... and I realized that I still hadn't fully understand the use cases of the html templates, the possible forms of the html templates and just as important, the form of the templates can have small variations that could cause the code to not work.

So here I was, spending a god chunk of a day trying to write code to optimize a process that I hadn't fully understood yet. And I have a feeling that some of the biggest failures of software projects is a poor understanding of the use cases which may need extensive revisions to deal with unforeseen problems.

More important than making it work quickly, is that it works first. I ought to stick to that before putting effort into speeding up certain processes. Once I get this templating engine up and running, it'll be interesting to see what uses I can come up with for this code.

Building a web craling platform

I've been doing some web-crawling over the last few years. I started off with some really rudimentary pattern matching stuff but over the last 2~3 years I've become much more comfortable with text processing. To the point where I am becoming able to programmatically to edit chunks of text that I write. The power of really good text processing is amazing, especially when it comes to structured textual information.

I have spent an innumerable amount of time trying to figure out the best way of writing programs to parse text and I have been thinking of programmatic methods of getting the data out the information out there. But the more and more time one spends with text, patterns and strategies arise that can be used over and over again to get at the information embedded in text. What I want to be able to do is to create a simple frame work that will allow me to quickly create parsers for whatever text document I want to get at and keep a library of them so I can stream data from a variety of websites. Eventually, I might be able to make it easy enough that even non-programmers can be able to write parsers too and that may have some interesting applications.

Running 4 km (almost) every weekday

As a result of the commute, I've been working on getting the most out of my time. Getting home by about 8~9 pm everyday means that I don't have a lot of time to do much else. Considering that I am now making time to study for my CFA exam in December and trying to exercise everyday, that doesn't leave me with much time to do anything else after getting home.

I've noticed that I've put on a little weight, about 3 kg compared to last year. I attribute that to getting a big bag of almond chocolates from Costco and having a kitchen where I am cooking much meatier dinners as of recent. So I am needing some cardiovascular exercises to act against the recent bad diet (which I will get back into making more healthy).

I have a transfer stop on the way home where I need to change trains which I need to ride for a single stop before getting home. I've decided to take this opportunity to skip taking the transfer and jogging the rest of the distance home. I've been cycling on and off since the last year, usually doing long treks on the weekend (about 30 km) and I was quite pleasantly surprised to find that exercise on the bike transferred quite nicely to running as running 4 km. Progressing through the first weekhop of running nearly everyday, I have found that my stamina has increased and that I can push up my speed significantly. There are also daily variations on my ability to run, however I am feeling an improving trend. I should start timing my runs to see how I progress.

By the time I get home after a quick hop into the shower, I've got time to pump some iron using the weight training machine I have in my room. I also have some free weights in my room that I use from time to time and I've noticed that using free weights are great for full body workouts instead of working on isolated muscles. I still am working on trying to figure out the perfect routine to really push my muscles. It looks like I'll either have to hit a gym sometimes and talk to a trainer or consult some books.

Conquering Excel Macros

VBA is a terrible language. It's archaic and idiosyncratic. I recently got back into programming in VBA since yesterday to help out a friend at work to process data faster. It would take him weeks to what this program will be able to do in minutes. The only kicker is that it costs me time to write this code.

I've transitioned over to Python as my main programming language about 4 years ago and I've learned a lot from it since starting. Functional programming and the map, reduce strategy to computing data. Basically, if you can write out a function to process one block of data, it's just a matter of looping through the rest of the data array to calculate everything.

What python (and several other languages)  do is obsolete the loop and make them implicit. If you have a function and you have an array, then there is way to apply the function to all the elements of an array with one line of code. None of that

• let's create counter,
• create the loop structure,
• create an output array,
• pass the data to the function,
• dump the output data into the array,
• and increment the counter to get to the next piece of data

process. You have a spoon and there are many buckets of ice cream, what else do you need to know? Functions and list comprehensions work just like that. In a single line of code that 6 step process is gone. Code that can comprehend lists are amazingly short and the hard thing to do it limited to writing the function.

I work in Python and I think in Python when I program. Then I come back to VBA, it doesn't have the syntax to do that. What usually took me 1 short line of code takes me 4~7 lines of longer code to do the thing. It's a waste of time and prone to error.

So basically, I've spent the last 2 days making VBA more Python like by creating the data structures to alleviate the lower level management of data array comprehension to the point where I can throw arrays around and process them in 2~3 lines of short code. It's not a perfect solution, but much better and far more manageable. Compared to the old procedural code that I used to write, managing old procedural code is a nightmare to maintain. The great thing about list  comprehension is that you can worry less about the state of the program because the code becomes stateless. None of that "what was the value of the counter?" and that pizzaz.

There are times when the state of the program is important however, like for instance a user application: What screen am I on, or what settings have I changed, or am I still connected to the internet or something like that. But still, the overhead of dealing with states associated with data greatly decreases with list comprehensions (there will be cases where the data will require state changes in a function, but the overhead is greatly reduced).

At the time, instead of having multiple lines to describe a process, you can have just one which is far easier to understand because the unimportant scaffolding is hidden; the only thing showing are the important parameters and the name of the function. That is how good code is written. So farewell to the terrible looping structure, I've gotten rid of most of them and now left with the descriptive code that tells me what it does and what it operates on.

That's the beauty of higher level computing languages, in that you're able to do a lot with saying a little. There are even higher level ones where you can define your own keywords and syntax to them. A not very well known language is Lisp that allows one to do just that. I have no idea what is possible when one is able to define one's own language to suit whatever problem they are tacking with but I am quite sure it would be a very fascinating adventure. I've already seen my programming skills improve greatly by using Python. I can only imagine what else is there to learn by learning higher level languages.

The mind is a lot fresher after spending time away

I have found that I need to take 2 passes at a problem before getting it right and usually the second pass needs to be done after spending a day away from the task. My mind feels significantly fresher after spending time away from work and then coming back to a task. Things that I didn't notice before just jumps out at me, where as in other cases, I could look over something repeatedly and still overlook details.

I find this to be true to both writing and even programming. The first post of anything I make tends to filled with errors and even after double checking, I still tend to miss a lot of small mistakes. It's likely related to a bad habit of wanting to get something done as fast as possible and sometimes I find double-checking as a tedious task that gets in the way. One hitting the submit button, that feeling of needing to have something done as quickly as possible usually subsides and I am able to look at past work with a more critical eye.

There are times when I have been coding to find myself coming back to the code wondering to myself "what the hell was I thinking" when a much simpler solution exists. This is especially true when I make the mistake of designing a complex solution and thinking of the solution as clever.

For example, I managed to write a text parser with the feature that it could automatically detect if a text string was an integer, a floating point number, exponential or a date and automatically call the correct function to parse the data. The function was also extensible in that it could also be updated on the fly to auto-detect and parse other values.

I thought the smart thing to do was to create 2 functions, 1 to detect if a text string was of a certain type (returning true or false) and then a second function to perform the conversion and I had to make an elaborate system to keep the testing function and the parsing function paired together which I thought was kind of unweildy, but I managed to do it. A day later, I realized that I could just have 1 function that would either return a value or nothing depending if the parse failed or passed and used that as the indicator if the right function had been called.

I could have been bashing at a problem through an entire day with an inelegant solution... and I deplore inelegant solutions because why work hard on a bad solution when you could be spending time on a better one instead? Thinking like this is both a blessing and a curse because the resuts of my work varies between "really good" or "none."

Anyways, hopefully I'll be getting back to posting on a more regular basis. I have more pockets of time where I am able to think compared to before and I hope for this to continue.

Less time online and improved concentration

I've been getting back in to studying for the CFA exam and one thing I noticed is how terrible my concentration was. Spending one's time reading online articles vs reading books is quite a different experience-- a large amount of information has been accumulated into a single place instead of scattered through a variety of webpages that one would have to usually have to search through. Having a good table of contents is also a boon to immediately pinpoint where in a document that you would need to go to get the information needed.

By having a large volume of high signal-to-noise information available, it significantly cuts down on the time required to search for information (which I think of as a distraction). The resulting effect is that having good books to do research from actually helps improves one's concentration instead of being distracted by looking for information (which may or may not be relevant).

One other interesting thing I've noticed about myself is that I tend to focus way better when I am working with other smart people. I believe that having the synergy of having a few smart people around to cover for information gaps or thought gaps cuts down distractions by the need to look for information. I believe that addictiveness of, say, video games in general can be attributed to having all of the information a player might need very accessible through a very intuitive interface or by having an environment where one can immediately figure things out with simple tests.

Right now, I think that the internet isn't an ideal place to learn in depth topics through websites as either the quality isn't there, finding good quality content is simply too much work or the information that you're looking for isn't covered in significant enough depth. I think that books and other resources fill those gaps.

One project that I would like to work on is to start creating a repository/network of high signal-to-noise sources and will be looking into a variety of tools to help me do that.

Cutting down on wasted time

I've been working on cutting down wasted time recently, considering that now with the commute that I leave the home at 7:30 am and don't get home till past 8:30 pm, I have come to the realization that I have little time to myself. It is paramount that I make the best of it so I've been working on cutting down the number of distractions and time killers.

One of the best moves I've made is killing the habit of flipping on the computer immediately after getting home. I don't have a TV and have been living without one for the past 5 years without one and to take its place, the computer has become my media center. The problem is that the way I am spending time on the computer is just as bad as the way I used to spend on TV-- aimlessly doing nothing.

Instead of flipping on the computer to aimless browse at things, am becoming more stringent with my time allocation. First thing I do after getting home is either cook or exercise. I've also started making a new habit of not riding the train all the way home and getting off at 1 stop before I get home. Actually, it's a transfer stop and instead of spending time waiting for my transfer, I just hit the street and jog the rest of the 3 km home. I've managed to do the jog in about 15~16 mins and it's turning into a pretty good exercise routine.

Running this distance is actually quite significant when it comes to reducing body fat as I burn about 200 calories. One average meal for me is about 500~600 calories so I am burning off the equivalent of 33%~40% of a single meal. Assuming that I do this 3~4 times a week (ie when it doesn't rain) the reduction in the amount of calories the body absorbs is significant in addition to the health benefits of jogging.

I've also allocated about 1 hour almost every night to studying for the CFA exam (at a rate of about 20 pages/day) and have made this a prerequisite to turning on the computer. It's been about 3 years since the last course I took in university and it's rather refreshing to start studying again. One of the great things about reading study material is the much higher signal-to-noise ratio compared to reading articles on the net. The quality of information in books is far superior to what is published on the net. I've also noticed that as much as I like digitized information, I find that I like writing out my notes; there is simply a freedom in penning out notes that isn't available compared to typing out information (writing diagrams and sketching arrows is still ridiculously slow). Now that I think about it, if there was a great way to pen notes digitally and organize them, that would be awesome because I am one of those people that can easily generate volumes of notes.

I wonder if there are good note taking software where you could both type and use a stylus to sketch in other information. Like arrows, lines and even equations.

Once getting into a groove of "getting things done in rapid succession," I've found that a momentum kicks in. In the sense that I am far more likely to want to move on to the next item on my mind that needs to get done, without hesitation. Looking back at past behavior, the biggest hindrance to action is thinking about all the things that I ought to be doing and not knowing which of the things I should just simply do, because I have a tendency of worrying about "is the thing I am doing the right thing to be doing?" I've moved on to killing that by coming to the realization that time is scarce and doing something is better doing nothing. I've become far more effecting at using my time when getting at home, far more compare to when I was living closer to work and I find that ironic.

A change in environment is a good thing.