I have a massive music collection, I just finished doing a scan and there are nearly 20,000 files sitting around on my hard drive. I used to be fairly good at organizing and maintaining musical playlists, keeping up with the sheer volume of music and the occasional need to move them around on the HD, it is hard keeping playlist files up-to-date.
I have been a fan of online services of Pandora (when it was available internationally) and last.fm to find new music and through them, I have found plenty of non-mainstream artists that I would have never found while listening to the radio while driving in the car; I have considerably expanded my music collection as a result.
Relational Databases: Eggs and Tomatoes so Cakes and Tomatoes?
Maintaining music playlists is sort of like making a meal, some dishes go great together and other dishes don't. This is probably a challenging problem since you don't necessarily want to be mixing classical music with hard rock (unless you have something creative in mind).
As musical libraries get large, the combinatorics dealing with making suitable playlists get semi-exponentially hard. It's actually an n! (n-factorial problem) for those of you that have studies advanced mathematics. It makes natural sense that a computer would be useful in dealing with these kinds of problems. This is precisely the kind of itch that online music recommendation systems such as Pandora and last.fm are trying to scratch from a different angle. What these guys are trying to do is recommend you music that you *don't* have that would make a great addition to your music selection. If you end up buying the song, then they get a cut and make some money. The kind of problem that I want to solve is given a library of music, automatically generate a set of music that I would like to listen to. I am willing to bet that these guys realized the problem that I wanted to solve far earlier than I did and are trying to turn it into an online business.
Unfortunately, Pandora and last.fm are recommendation systems not services used to manage your musical collection. The next best thing that I've found so far is iTunes' "Genius," which I am experimenting with right now.
Though I do not know the exact mechanics of how "Genius" works, I do know that the system reads the ID3 tags of the song, giving album name, artist, genre and other information to make a guess as to what other songs that could be used to make a good playlist.
Without knowing anything about the science of playlist generation, I can propose at least one technique. The first is to use a Bayesian filter to as a first pass at a music playlist generator. If you can get access to the sequence of music that a person plays you could calculate the P(A|B) which means the probability of having song "A" played given that song "B" was played. Using this technique you could make a crude filtering mechanism. The interesting thing about this technique is that it is already used in spam filtering of e-mail by making a database of words that usually occur in a non-spam e-mail and comparing that against e-mails that comes into your mailbox. If the mail doesn't fit your "taste" it generally is labeled as spam and removed. Other techniques such as say a Markov chain is a variation of the Bayes filtering technique.
More advanced techniques actually analyze the song itself, identifying musical instruments, beats and style. I consider this a "hard problem" and would not want to touch this, but people from the Music Genome Project have been working on this problem, which was the kernel that gave birth to the Pandora online music service.
What I don't like about iTunes
After giving Genius a spin, I do like it. It's not 100% perfect but it is good at making playlists of standard music genres: Pop, Alternative/Rock, BritPop, House/Electornica and other fairly common mixes, however there it does miss some less common genres, like break-beat Jazz/Funk/Jazz-electronica kinds of mixes that I enjoy. There is also the problem that I would like to keep some music genres out of some mixes, say for Downbeat House/Electronica, I generally like to keep hard Trance/Techno out of the collection. But given a playlist, I can probably make minor tweaks to make it acceptable.
The other thing about iTunes I don't like is that it is slow and clunky-- it isn't as snappy as compared to the traditional music players like winamp I've used on Windows. I've also started using programs to start adding and fixing ID3 music information to match up with data stored in online CD databases online, which should make the auto playlist generation more effective. The programs I've been using are handy that they can also be used to automatically rename files to any format I like -> meaning if I had an album saved just as "track 01, track 02" it could be converted to "track number - artist - song name." I have done this with a lot of the data I have already, but the problem is that after renaming these files, file links in Genius get broken. Putting the renamed filenames into the database is easy, but the old entries still remain and I have had to do removal manually, which is a pain in the ass. Still looking for alternatives, but it's been an interesting journey into the world of automatically generated musical playlists.
Monday, March 29, 2010
Saturday, March 27, 2010
Too much software/technology a bad thing?
I caught a cold starting last week. I took a day off of work on Thursday to head the cold off, unfortunately it got worse on Friday and I had work do be done. So instead, I am spending the weekend recovering. I can only sleep so much and since I can't go out and play, I am stuck at home awake with nothing to do. So I've had a lot of time to kill with writing.
This post comes off as a continuation of Sacha's post "Hacking the iPhone was inevitable." The iPhone and any electronic device running software is probably hackable. How far people go to hack hardware is deep.
I managed to meet the coordinator of CanSecWest before through some interesting connections. I regret that I am bad at keeping up with people so I am not in as close contact anymore with these guys. But there are people out there that know how to take the packaging off of a chip and analyze the memory contents of a chip using an scanning electron microscope. There are other neat hacks where code protection on some microprocessors can be disabled by heating up the chip to cause some bits to flip and the data to come out. Who knows what creative avenues hackers have to get at software and hardware.
The risks of getting hacked is now pervasive. There are now services in the US used to disable cars if people miss their car loan payments. A disgruntled former employee managed to deactivate 100 cars after being fired after remotely logging into his company's network.
So should we feel safe with software handling data or mechanical systems?
The biggest benefit and the biggest danger of software is its power to scale with its user base. It's only marginally more expensive for a computer to handle data for 100 people vs 1,000,000 people with the right infrastructure. The power for software to scale is what provides the capacity for our modern systems to perform work for us. If 1 server handles data for 100 people and gets taken down then it isn't a huge problem, but what if a server handling 1,000,000? The negative effects are also multiplied by scaling.
What we have here is a potential "black swan" kind of event where a usually reliable system (99.999%) breaks down just once over 4~5 years causing catastrophic damage. The black swan effect is what is used to describe what happened in the the recent US stock market crash, it can also be used to describe other events such as Chernobyl or the huge power outage that happened in Eastern North America many years ago. What modern society right now is in effect doing it putting all of its eggs into an electronic basket and we are expecting it to be very reliable.
A secret weapon out there that could selectively kill electronic devices in some huge blast radius (ie. War of the Worlds style) would be crippling, like a torpedo up the exhaust pipe of the Death Star.
Deciding how far we should rely on technology is an interesting question and the only best answer I can give is "what and how much are you willing to risk?" That is... if you know that you are taking a risk in the first place.
This post comes off as a continuation of Sacha's post "Hacking the iPhone was inevitable." The iPhone and any electronic device running software is probably hackable. How far people go to hack hardware is deep.
I managed to meet the coordinator of CanSecWest before through some interesting connections. I regret that I am bad at keeping up with people so I am not in as close contact anymore with these guys. But there are people out there that know how to take the packaging off of a chip and analyze the memory contents of a chip using an scanning electron microscope. There are other neat hacks where code protection on some microprocessors can be disabled by heating up the chip to cause some bits to flip and the data to come out. Who knows what creative avenues hackers have to get at software and hardware.
The risks of getting hacked is now pervasive. There are now services in the US used to disable cars if people miss their car loan payments. A disgruntled former employee managed to deactivate 100 cars after being fired after remotely logging into his company's network.
So should we feel safe with software handling data or mechanical systems?
The biggest benefit and the biggest danger of software is its power to scale with its user base. It's only marginally more expensive for a computer to handle data for 100 people vs 1,000,000 people with the right infrastructure. The power for software to scale is what provides the capacity for our modern systems to perform work for us. If 1 server handles data for 100 people and gets taken down then it isn't a huge problem, but what if a server handling 1,000,000? The negative effects are also multiplied by scaling.
What we have here is a potential "black swan" kind of event where a usually reliable system (99.999%) breaks down just once over 4~5 years causing catastrophic damage. The black swan effect is what is used to describe what happened in the the recent US stock market crash, it can also be used to describe other events such as Chernobyl or the huge power outage that happened in Eastern North America many years ago. What modern society right now is in effect doing it putting all of its eggs into an electronic basket and we are expecting it to be very reliable.
A secret weapon out there that could selectively kill electronic devices in some huge blast radius (ie. War of the Worlds style) would be crippling, like a torpedo up the exhaust pipe of the Death Star.
Deciding how far we should rely on technology is an interesting question and the only best answer I can give is "what and how much are you willing to risk?" That is... if you know that you are taking a risk in the first place.
Investing in and Filtering for People
The thing about moving about moving away from home is that it puts you out of reach of your old social network. It gets harder to stay in touch and interact with distance. And thus, one ends up making new friends and a new social network.
The thing about living as an expat abroad, most of the people I have close relations with are generally expats themselves. Living in a different country changes people, in the way they deal with a new culture and find acceptance (or reject it) for what it is. I have a great deal easier of a time relating to people that have lived abroad because of common experiences.
There are many quirks about Japanese culture that doesn't jive with me, these are broad generalizations of course... for example the Japanese tend to be shy about meeting other Japan. They run events differently here, in western culture if you have a home party, usually you just have invite people over and the introductions happen on their own. In Japan, that doesn't usually happen and depending on the event there is time set a time to introduce one's self to the entire group where there is one speaker and everyone else is speaking.
There are actually "group date dinners" where a girl invites her female friends out and a guy invites his guy friends out and they have dinner together to figure out who might get along with who. The whole notion of having a party where the focus is to "meet" people is artificial to me. So I've generally stayed out of those things, but I digress.
Some parts of their hierarchical social structure also doesn't fit with me since younger people are usually expected to be somewhat sub-servant to their elders in a semi-formal environment, even if they are just 1 or 2 years apart. They use their last names at work or at university and it gets confusing in a social setting since Japanese people use first names in a non-formal situations. However, even in a social setting, work or university related people still end up using their last names. This makes it a pain in the ass sometimes mixing work with one's social life. Then again, I am digressing again and this post is about social networks.
The problem about living as a foreigner in Japan is that the foreigner community is transient, everyone is coming and going. In about a year, you might end up losing half of your friends since they go back home or somewhere else. Not that it is a bad thing, because having friends around the world is a great thing. Most of the time, you just end up meeting new people to do things with.
Usually within the international community, the easiest way to meet people is through some event held by an expat. Western culture rules apply here and it is quite easy to meet and make friends from there, what happens is up to you. Most friends I do make is usually the result of these events and we stay in contact and call each other out when another event happens.
But when you get older, the idea of just hanging out gets old. When I was younger it was sort of the like that for me, and I would sometimes wonder why I was at some social event just bantering with people, because at the end of the day, it was just socializing.
A purposeful relationship with someone is being able to do something constructive together. Constructive could be teaching each other new skills, exchanging information, working on projects together, introducing each other to new resources or something along those lines. As generalized as it sounds, specific examples could be having someone to host a party with, passing each other information on new events that might be interesting, checking out a great restaurant that one of them might have heard off. I mentioned previously in a post talking about people as information filters.
Not only can we consider a network of "good" friends as a network of good information filters, they also form a symbiotic network supporting all nodes in the network. I believe that this was one of the key elements in the rise of social networks; the "internet 2.0" buzzword that we haven't been hearing from as of recent.
I believe that social networks have been on a decline as of recent. Many of the most popular community sites and social network pages have been losing value. I would argue that the potency of a social network is diluted as more and more people join, unless there is a mechanism to filter people from entering a network.
Sites that I can name off the top of my head that have become victims of this phenomena are Digg and Reddit, both are social bookmarking-esque sites, where people would submit links of interest and the most popular ones would be voted up. I never followed digg, but I was with Reddit since it was in its early phases, before it got popular. The quality of the links posted were really good, since the community was small and driven by fairly intellectual people who probably learned of this site through a startup network (Reddit was a startup at first). Reddit links used to consist of a set of links to startup, economic and international related news. I found this site to be very educational in its early stage.
After several years and growing popularity, mainstream internet users started using the site and more general links were being submitted and as a result the quality of links (relative to me) started to decline as I didn't care much of the latest internet meme, funny picture or what not. The power of the old community was eventually diluted by mainstream internet users.
Fortunately, reddit started to create sub-groups which has started to filter people by field of interest. Though I think there are still too many people on the site. Reddit is also a good driver of traffic so I would also expect people with financial interest in popularizing their sites be gaming the system to make their links popular by resubmission and other underhanded tactics. Thus the quality of social networking sites declines with increasing user base.
I believe that there was an argument that "the collective IQ of a group decreases with more people" and this is probably what is in play here. Knowing how to maintain value in a social network requires skill in isolating the valuable parts and blocking off the rest of the noise.
This applies as much to the online world as it does to real life. The problem I face is social capital dilution-- because most of my meetings with people have been at large events. This problem is amplified even more after coming to Toyko the people I know are scattered all over the place, making it harder to meet on a regular basis, thus making large parties the most "efficient" way of catching up with everyone.
The end result is that I need to become more active in being more selective in the people I meet and for what purpose. This will be an interesting thought to try.
The thing about living as an expat abroad, most of the people I have close relations with are generally expats themselves. Living in a different country changes people, in the way they deal with a new culture and find acceptance (or reject it) for what it is. I have a great deal easier of a time relating to people that have lived abroad because of common experiences.
There are many quirks about Japanese culture that doesn't jive with me, these are broad generalizations of course... for example the Japanese tend to be shy about meeting other Japan. They run events differently here, in western culture if you have a home party, usually you just have invite people over and the introductions happen on their own. In Japan, that doesn't usually happen and depending on the event there is time set a time to introduce one's self to the entire group where there is one speaker and everyone else is speaking.
There are actually "group date dinners" where a girl invites her female friends out and a guy invites his guy friends out and they have dinner together to figure out who might get along with who. The whole notion of having a party where the focus is to "meet" people is artificial to me. So I've generally stayed out of those things, but I digress.
Some parts of their hierarchical social structure also doesn't fit with me since younger people are usually expected to be somewhat sub-servant to their elders in a semi-formal environment, even if they are just 1 or 2 years apart. They use their last names at work or at university and it gets confusing in a social setting since Japanese people use first names in a non-formal situations. However, even in a social setting, work or university related people still end up using their last names. This makes it a pain in the ass sometimes mixing work with one's social life. Then again, I am digressing again and this post is about social networks.
The problem about living as a foreigner in Japan is that the foreigner community is transient, everyone is coming and going. In about a year, you might end up losing half of your friends since they go back home or somewhere else. Not that it is a bad thing, because having friends around the world is a great thing. Most of the time, you just end up meeting new people to do things with.
Usually within the international community, the easiest way to meet people is through some event held by an expat. Western culture rules apply here and it is quite easy to meet and make friends from there, what happens is up to you. Most friends I do make is usually the result of these events and we stay in contact and call each other out when another event happens.
But when you get older, the idea of just hanging out gets old. When I was younger it was sort of the like that for me, and I would sometimes wonder why I was at some social event just bantering with people, because at the end of the day, it was just socializing.
A purposeful relationship with someone is being able to do something constructive together. Constructive could be teaching each other new skills, exchanging information, working on projects together, introducing each other to new resources or something along those lines. As generalized as it sounds, specific examples could be having someone to host a party with, passing each other information on new events that might be interesting, checking out a great restaurant that one of them might have heard off. I mentioned previously in a post talking about people as information filters.
Not only can we consider a network of "good" friends as a network of good information filters, they also form a symbiotic network supporting all nodes in the network. I believe that this was one of the key elements in the rise of social networks; the "internet 2.0" buzzword that we haven't been hearing from as of recent.
I believe that social networks have been on a decline as of recent. Many of the most popular community sites and social network pages have been losing value. I would argue that the potency of a social network is diluted as more and more people join, unless there is a mechanism to filter people from entering a network.
Sites that I can name off the top of my head that have become victims of this phenomena are Digg and Reddit, both are social bookmarking-esque sites, where people would submit links of interest and the most popular ones would be voted up. I never followed digg, but I was with Reddit since it was in its early phases, before it got popular. The quality of the links posted were really good, since the community was small and driven by fairly intellectual people who probably learned of this site through a startup network (Reddit was a startup at first). Reddit links used to consist of a set of links to startup, economic and international related news. I found this site to be very educational in its early stage.
After several years and growing popularity, mainstream internet users started using the site and more general links were being submitted and as a result the quality of links (relative to me) started to decline as I didn't care much of the latest internet meme, funny picture or what not. The power of the old community was eventually diluted by mainstream internet users.
Fortunately, reddit started to create sub-groups which has started to filter people by field of interest. Though I think there are still too many people on the site. Reddit is also a good driver of traffic so I would also expect people with financial interest in popularizing their sites be gaming the system to make their links popular by resubmission and other underhanded tactics. Thus the quality of social networking sites declines with increasing user base.
I believe that there was an argument that "the collective IQ of a group decreases with more people" and this is probably what is in play here. Knowing how to maintain value in a social network requires skill in isolating the valuable parts and blocking off the rest of the noise.
This applies as much to the online world as it does to real life. The problem I face is social capital dilution-- because most of my meetings with people have been at large events. This problem is amplified even more after coming to Toyko the people I know are scattered all over the place, making it harder to meet on a regular basis, thus making large parties the most "efficient" way of catching up with everyone.
The end result is that I need to become more active in being more selective in the people I meet and for what purpose. This will be an interesting thought to try.
Friday, March 26, 2010
Reinveting one's self
I am sick, pumped up on medicine and drinking a home brew fruit punch with sake mixed in. Suffice to say that my thoughts are a little more fluid and out of control as usual, but I will take it as a good thing, because it is a good thing to let loose once in a while.
If someone asked me what my biggest weakness was, I'd have to say that I am too accommodating. There is an innate nature (whether if it is genetic or nurtured) that expects myself to fulfill the expectations of other people. In general, I think it is a general Asian weakness, more so a Japanese one? A Chinese one? I am not sure, but it is something that most of us Asian kids have to end up dealing with --> dealing with the academic expectations of our parents. Most of us that do well academically are the ones that know how to jump the hoops and please our teachers and parents. But I gotta say, the ones that only know how to jump the hoops and play the tricks end up turning into a bunch of wusses.
I believe it because I think I am one.
By most standards, I would consider myself largely successful. I got accepted into a pretty good university and graduated out of one of their most demanding undergraduate departments, with distinction. UBC is ranked in the 35th out of all the universities in the world.
I would eventually get a shit job cutting out clouds from satellite images and then proceeded to look for other opportunities. I eventually would find a scholarship to Japan, with all university expenses paid and a monthly living allowance. I would be 1 out of 13 students accepted into this program to head to Japan and land into one of their top 5 universities for Engineering and one of the best labs for Microsystems research. I would develop and publish research results for an electron beam gun with the lowest turn on voltage published in literature I could find at that time.
By most standards, yeah I guess I am successful, but I did a lot of this by giving people what they wanted. I followed the rules and played by the book, produced what I had to do. Looking back at it, it might feel like being a dog knowing to do tricks.
As a kid, I used to play music and had to audition to get into bands. It's kind of interesting to realize that by the age of 13 I was learning interviewing skills and started figuring out the kinds of answers these interviewers were looking for. It's a system to be gamed, really. All you had to know is what the person in front of you wants and you give it to them. I've sent a friend on a 1 year scholarship exchange to France because I knew how to set her application up right and I am doing it again to land her a job in a research institute in France.
And that's the problem with the thing. I've been trained like a monkey to produce for the demands of other people. I wouldn't consider it completely a bad thing, but it's unbalanced. Why?
Where is the "me" in this? Where is the "I" in this fucking equation? And that is the problem.
I've been mulling over a section from the book The Art of Seduction I've mostly finished reading but got bored because it was a repeat of a similar story over and over again. Basically the author generalizes "seducers" or character archetypes into several categories and provides a sort description of their strengths and weaknesses.
From the book, I could describe myself from the "The Charismatic" archetype, a person that knows the right things to say and attains a level of popularity from it. I work well in public environments and would consider myself an extrovert, well, that was while I was in Canada. The human dynamics of people between Japanese and Canadian culture is different so my skills don't entirely apply here so I am far more introverted in Japan. But, continuing on...
As the book would go on to describe, the greatest weakness of a charismatic person has is with dealing with people that simply don't give a damn. The book describes these people as "the rake" but I feel that "rogue" would be a better title for a person with these qualities.
These bastards know only what they want and are out to get what they want. They don't care to be pleased and if you are in their way, prepared to get removed unless you put up a fight.
The last time I got pushed over by one of these guys was when they were after a girl I met at a party. Sure they were friendly when we weren't "competing" when we had conflicting interests, but when it came down to their priority of getting the girl, "superficial friendship" was something easy to toss for them. No amount of charm, discussion or whatever was going to convince them otherwise. I have a sinking feeling that in the higher echelons of politics and business are polluted with these people. For these people are Alpha, people that use and wield power for their own purposes. The accommodating nice guy/person would generally get their asses handed to them because of these people.
Many a movie has been made with this plot line.
Revenge of the Nerds
There is a joke out there that went something like this...
"I might be ugly but you're stupid, but at least there's plastic surgery"
I believe that I've only been playing with half a hand of cards. One half is knowing how to please people and gain support through that, the other half is knowing how to go at it on your own if you have to. There are some serious assholes out there that play with the opposite hand that I carry. The ideal would be a combination of both.
I've lived in Sendai for about 3 years and moved to Tokyo. Tokyo and Sendai are relatively polite places, but move out further west to Osaka, people are way more self-centric, which may appear to be very un-Japanese. Even the Japanese consider western Japan to be almost another country because of the big differences in thinking. I've had to deal with some of them on the phone and I can already tell the difference.
Unfortunately for me is that I've been living in the accommodating parts of Japan for a little to long and it's had a bad impact on me. This is going to have to change and I am going to start aiming to become more assertive, but without crossing the line of becoming an ass. When asserted, I want to command respect when I speak.
When I was younger, I made a decision that I didn't want to be "cool" because I didn't like cool people when I was younger. They seemed shallow, relying on their social status as a source of power and popularity. People tried to please them and their approval would be like something of a boss giving someone a promotion. They had the power and they didn't have to give a damn. I didn't care for being judged by other people like this and stuck to my own world of whatever the hell it was I existed in.
When the era of the jockeys and "cool" people ended from grade school as we walked into university, the playing field was considerably leveled and I wasn't as limited in my options anymore. I chose not to become "cool" because of my distaste for it and I disliked the notion of attaining social power because I felt it was corrupting. However, there are things that I realize I am at a loss for, such as knowing how to handle power if given it in the first place.
In the work place, I am sort of like a boss to several people that work under me, and I am starting to realize that my ability to give orders is somewhat undermined by my inability to take a controlling position as I have a tendency to treat people as equals. In other words, I might be their boss, but they are less willing to take orders from me, especially when it comes to doing tedious but necessary tasks. As much as I might hate it, some parts of being a good leader is knowing how to handle power and give orders.
My vision of leadership was never like this before because I believed in working with people with similar level of skill and leadership would be diplomatic coordination or a person with a vision that everyone else believed in. Not simply giving and taking orders... and so is my welcome to the corporate world.
The world is more diverse than I first realized because of the bubble I existed in. I've slowly stepped out of it and will need to shift my perceptions and acquire new skills to deal with it.
If someone asked me what my biggest weakness was, I'd have to say that I am too accommodating. There is an innate nature (whether if it is genetic or nurtured) that expects myself to fulfill the expectations of other people. In general, I think it is a general Asian weakness, more so a Japanese one? A Chinese one? I am not sure, but it is something that most of us Asian kids have to end up dealing with --> dealing with the academic expectations of our parents. Most of us that do well academically are the ones that know how to jump the hoops and please our teachers and parents. But I gotta say, the ones that only know how to jump the hoops and play the tricks end up turning into a bunch of wusses.
I believe it because I think I am one.
By most standards, I would consider myself largely successful. I got accepted into a pretty good university and graduated out of one of their most demanding undergraduate departments, with distinction. UBC is ranked in the 35th out of all the universities in the world.
I would eventually get a shit job cutting out clouds from satellite images and then proceeded to look for other opportunities. I eventually would find a scholarship to Japan, with all university expenses paid and a monthly living allowance. I would be 1 out of 13 students accepted into this program to head to Japan and land into one of their top 5 universities for Engineering and one of the best labs for Microsystems research. I would develop and publish research results for an electron beam gun with the lowest turn on voltage published in literature I could find at that time.
By most standards, yeah I guess I am successful, but I did a lot of this by giving people what they wanted. I followed the rules and played by the book, produced what I had to do. Looking back at it, it might feel like being a dog knowing to do tricks.
As a kid, I used to play music and had to audition to get into bands. It's kind of interesting to realize that by the age of 13 I was learning interviewing skills and started figuring out the kinds of answers these interviewers were looking for. It's a system to be gamed, really. All you had to know is what the person in front of you wants and you give it to them. I've sent a friend on a 1 year scholarship exchange to France because I knew how to set her application up right and I am doing it again to land her a job in a research institute in France.
And that's the problem with the thing. I've been trained like a monkey to produce for the demands of other people. I wouldn't consider it completely a bad thing, but it's unbalanced. Why?
Where is the "me" in this? Where is the "I" in this fucking equation? And that is the problem.
I've been mulling over a section from the book The Art of Seduction I've mostly finished reading but got bored because it was a repeat of a similar story over and over again. Basically the author generalizes "seducers" or character archetypes into several categories and provides a sort description of their strengths and weaknesses.
From the book, I could describe myself from the "The Charismatic" archetype, a person that knows the right things to say and attains a level of popularity from it. I work well in public environments and would consider myself an extrovert, well, that was while I was in Canada. The human dynamics of people between Japanese and Canadian culture is different so my skills don't entirely apply here so I am far more introverted in Japan. But, continuing on...
As the book would go on to describe, the greatest weakness of a charismatic person has is with dealing with people that simply don't give a damn. The book describes these people as "the rake" but I feel that "rogue" would be a better title for a person with these qualities.
These bastards know only what they want and are out to get what they want. They don't care to be pleased and if you are in their way, prepared to get removed unless you put up a fight.
The last time I got pushed over by one of these guys was when they were after a girl I met at a party. Sure they were friendly when we weren't "competing" when we had conflicting interests, but when it came down to their priority of getting the girl, "superficial friendship" was something easy to toss for them. No amount of charm, discussion or whatever was going to convince them otherwise. I have a sinking feeling that in the higher echelons of politics and business are polluted with these people. For these people are Alpha, people that use and wield power for their own purposes. The accommodating nice guy/person would generally get their asses handed to them because of these people.
Many a movie has been made with this plot line.
Revenge of the Nerds
There is a joke out there that went something like this...
"I might be ugly but you're stupid, but at least there's plastic surgery"
I believe that I've only been playing with half a hand of cards. One half is knowing how to please people and gain support through that, the other half is knowing how to go at it on your own if you have to. There are some serious assholes out there that play with the opposite hand that I carry. The ideal would be a combination of both.
I've lived in Sendai for about 3 years and moved to Tokyo. Tokyo and Sendai are relatively polite places, but move out further west to Osaka, people are way more self-centric, which may appear to be very un-Japanese. Even the Japanese consider western Japan to be almost another country because of the big differences in thinking. I've had to deal with some of them on the phone and I can already tell the difference.
Unfortunately for me is that I've been living in the accommodating parts of Japan for a little to long and it's had a bad impact on me. This is going to have to change and I am going to start aiming to become more assertive, but without crossing the line of becoming an ass. When asserted, I want to command respect when I speak.
When I was younger, I made a decision that I didn't want to be "cool" because I didn't like cool people when I was younger. They seemed shallow, relying on their social status as a source of power and popularity. People tried to please them and their approval would be like something of a boss giving someone a promotion. They had the power and they didn't have to give a damn. I didn't care for being judged by other people like this and stuck to my own world of whatever the hell it was I existed in.
When the era of the jockeys and "cool" people ended from grade school as we walked into university, the playing field was considerably leveled and I wasn't as limited in my options anymore. I chose not to become "cool" because of my distaste for it and I disliked the notion of attaining social power because I felt it was corrupting. However, there are things that I realize I am at a loss for, such as knowing how to handle power if given it in the first place.
In the work place, I am sort of like a boss to several people that work under me, and I am starting to realize that my ability to give orders is somewhat undermined by my inability to take a controlling position as I have a tendency to treat people as equals. In other words, I might be their boss, but they are less willing to take orders from me, especially when it comes to doing tedious but necessary tasks. As much as I might hate it, some parts of being a good leader is knowing how to handle power and give orders.
My vision of leadership was never like this before because I believed in working with people with similar level of skill and leadership would be diplomatic coordination or a person with a vision that everyone else believed in. Not simply giving and taking orders... and so is my welcome to the corporate world.
The world is more diverse than I first realized because of the bubble I existed in. I've slowly stepped out of it and will need to shift my perceptions and acquire new skills to deal with it.
Wednesday, March 24, 2010
The power of writing
Many people underestimate the power of being a good writer. As the age old adage goes, "the pen is mightier than the sword" and it is just as true now as it was first uttered.
I am now helping a Japanese friend on writing her resume and fortunate for her, she is an incredibly accomplished person (though she doesn't realize it), holder of several scholarships, awards and active in a variety of community projects, I was sent an incredibly sad looking resume saying something to the effect of "I am a hard worker and I would like to apply to this foreign position."
Apparently she also blurred the concept of a resume with a cover letter which weakened things considerably. Participating in the COOP program in university honestly paid off in spades as I am very aware of how resumes are read and what should be highlighted. The first version of the resume was immediately trashed and I've highlighted all of her special achievements and important contributions/results as these are the things that as an interviewer exactly wants to know; what makes you standout compared to everyone.
It's one thing to have good content and it is another to know how to present it well, but having both is very important. I've already managed to get her a 1 year scholarship to France by fixing her cover letter (and her prof's recommendation letter) to make a strong application package and hopefully I'll have have her accepted into some research company abroad if things work out. Knowing that I can pull these things off for other people is kind of fun. I just wonder what kind of value would a good technical writing market for students or Japanese people dealing with foreign relations is out there because having a good editor/writer in their pocket can make a huge difference (and we all know about Japanese "Engrish").
I should be glad that I'm practiced at writing application letters, I guess I did learn something from University after all.
I am now helping a Japanese friend on writing her resume and fortunate for her, she is an incredibly accomplished person (though she doesn't realize it), holder of several scholarships, awards and active in a variety of community projects, I was sent an incredibly sad looking resume saying something to the effect of "I am a hard worker and I would like to apply to this foreign position."
Apparently she also blurred the concept of a resume with a cover letter which weakened things considerably. Participating in the COOP program in university honestly paid off in spades as I am very aware of how resumes are read and what should be highlighted. The first version of the resume was immediately trashed and I've highlighted all of her special achievements and important contributions/results as these are the things that as an interviewer exactly wants to know; what makes you standout compared to everyone.
It's one thing to have good content and it is another to know how to present it well, but having both is very important. I've already managed to get her a 1 year scholarship to France by fixing her cover letter (and her prof's recommendation letter) to make a strong application package and hopefully I'll have have her accepted into some research company abroad if things work out. Knowing that I can pull these things off for other people is kind of fun. I just wonder what kind of value would a good technical writing market for students or Japanese people dealing with foreign relations is out there because having a good editor/writer in their pocket can make a huge difference (and we all know about Japanese "Engrish").
I should be glad that I'm practiced at writing application letters, I guess I did learn something from University after all.
Monday, March 22, 2010
I am terrible at shopping for clothes
I usually do my shopping from a utilitarian + price mizer point of view, if it can get the job done and it's cheap then I'll buy it.
This philisophy when it comes to shopping has probably saved me vast amounts of money as I am either certain about my needs or what I can live without and aim for something that fits in an "acceptable" price target.
I feel that I have made many good decisions in the past where I have decided against buying a car or living with roommates in a shared apartment and saving cash to use for other purposes. When it comes to shopping for things that lie outside of the world of price/performance comparison I might be a total cluts.
The company dorm which I live in, has communal washer and driers that everyone uses. I don't know exactly how, but I have had several pants get tears in them in very conspicuous places, namely in the butt (and no, it isn't me getting fat, really).
With me running out of wearable pants to wear, I've decided to head out to do some clothing shopping today, which is something that I rarely do. I wear most of my clothes until raggedy and then contemplate on buying something new. Generally, the only time I do buy clothes is when I make a visit back to Canada since I don't understand Japanese fashion too much and I like plain looking clothing. I do keep some nice clothing around for special events, however, but I am going on a tangent.
There are at least 2 points of weaknesses for me when it comes to shopping for clothes in Japan. First is that I don't know the brands or the stores out here all that well. I know just maybe just 3 by name and that is about it and the second is that given 2 pairs of pants, one of them being designer and a no-name brand. I generally opt for the no-name brand so long as I like the style and it is comfortable.
As of recent, I have had several of my female friends mention that I would look way better if I bought some nicer/fancier clothing. I have decided that it would be interesting to see what would happen if I did end up trying a few new styles.
In general, the price target for pants for me is in the $20~$30 range and generally the same things for sweaters. Only in rare cases when I do really like something that I will consider going upwards to maybe $45~$60 for a shirt, but that is only in rare cases.
The thing is that given 2 shirts, when the both of them provide for similar functionality (as in keeping me warm/clothes/or from breaking public nudity laws) I don't see a huge difference between them, I might like one style a little more than another but I generally can't justify myself paying close to 2x or even 3~4x in some cases for fabric that has been cut differently. Is it because it's been cut and put together differently that it is say 2x~3x more good looking as the price implies? What about the differences in material, is there that big of a difference?
To be honest, I have nearly no clue when it comes to clothing as my purchasing decisions come down to:
1. Is it the right price?
2. Does it look fashionably acceptable to me?
3. Is it comfortable?
I spent an entire afternoon going around to different shops looking at clothing. After spending 4 hours of looking around, I managed to buy a single pair of pants and that was about it.
I also checked out some designer looking stores and the one thing that struck me was that the clothing sold there is amazingly expensive. I would not fathom myself spending $120 for a pair of pants, $80 for a shirt and maybe $150 for a jacket. That would easily put the price tag of a single set of clothing at about $450 and this is only a single set of clothing we are talking about here! Sure the material looks a little nicer, but to me I just can't justify the price. For that kind of money, I'd rather save it and spend it on traveling or something else.
For the time being though, I am going to try and read up on clothing to try and figure out why there might be such a big price difference. A micro-fiber t-shirt vs a cotton one, sure I might be able to understand that (I love my micro-fiber shirt for sports and outdoor activities) but for other stuff, I haven't a clue.
This philisophy when it comes to shopping has probably saved me vast amounts of money as I am either certain about my needs or what I can live without and aim for something that fits in an "acceptable" price target.
I feel that I have made many good decisions in the past where I have decided against buying a car or living with roommates in a shared apartment and saving cash to use for other purposes. When it comes to shopping for things that lie outside of the world of price/performance comparison I might be a total cluts.
The company dorm which I live in, has communal washer and driers that everyone uses. I don't know exactly how, but I have had several pants get tears in them in very conspicuous places, namely in the butt (and no, it isn't me getting fat, really).
With me running out of wearable pants to wear, I've decided to head out to do some clothing shopping today, which is something that I rarely do. I wear most of my clothes until raggedy and then contemplate on buying something new. Generally, the only time I do buy clothes is when I make a visit back to Canada since I don't understand Japanese fashion too much and I like plain looking clothing. I do keep some nice clothing around for special events, however, but I am going on a tangent.
There are at least 2 points of weaknesses for me when it comes to shopping for clothes in Japan. First is that I don't know the brands or the stores out here all that well. I know just maybe just 3 by name and that is about it and the second is that given 2 pairs of pants, one of them being designer and a no-name brand. I generally opt for the no-name brand so long as I like the style and it is comfortable.
As of recent, I have had several of my female friends mention that I would look way better if I bought some nicer/fancier clothing. I have decided that it would be interesting to see what would happen if I did end up trying a few new styles.
In general, the price target for pants for me is in the $20~$30 range and generally the same things for sweaters. Only in rare cases when I do really like something that I will consider going upwards to maybe $45~$60 for a shirt, but that is only in rare cases.
The thing is that given 2 shirts, when the both of them provide for similar functionality (as in keeping me warm/clothes/or from breaking public nudity laws) I don't see a huge difference between them, I might like one style a little more than another but I generally can't justify myself paying close to 2x or even 3~4x in some cases for fabric that has been cut differently. Is it because it's been cut and put together differently that it is say 2x~3x more good looking as the price implies? What about the differences in material, is there that big of a difference?
To be honest, I have nearly no clue when it comes to clothing as my purchasing decisions come down to:
1. Is it the right price?
2. Does it look fashionably acceptable to me?
3. Is it comfortable?
I spent an entire afternoon going around to different shops looking at clothing. After spending 4 hours of looking around, I managed to buy a single pair of pants and that was about it.
I also checked out some designer looking stores and the one thing that struck me was that the clothing sold there is amazingly expensive. I would not fathom myself spending $120 for a pair of pants, $80 for a shirt and maybe $150 for a jacket. That would easily put the price tag of a single set of clothing at about $450 and this is only a single set of clothing we are talking about here! Sure the material looks a little nicer, but to me I just can't justify the price. For that kind of money, I'd rather save it and spend it on traveling or something else.
For the time being though, I am going to try and read up on clothing to try and figure out why there might be such a big price difference. A micro-fiber t-shirt vs a cotton one, sure I might be able to understand that (I love my micro-fiber shirt for sports and outdoor activities) but for other stuff, I haven't a clue.
Momentum
"An object at rest tends to stay at rest and an object in motion tends to stay in motion with the same speed and in the same direction unless acted upon by an unbalanced force." - Newton's first law of motion
As with moving bodies, newton's first law also seems to apply to life in motion; to those that are active, to those in inaction, or to those that need a change in vector.
One thing that I miss is being around creative technologists- people that like to do interesting things with technology. For the work environment I exist in, I just haven't been seeing the level of creativity that I want to see. Maybe it's the maturity level of the project that I've been working on but still, I haven't been able to take part to produce something with some level of personal creative satisfaction.
Some of the most fun projects I've worked on were simple things on my off hours with friends or on my own. More of like a dabbler but I would like to dedicate more time to learning how to do certain things with a professional flare maybe sometimes in the future. But usually, most of the best ideas I've seen have come out from people working in their garage to scratch some kind of itch, whether be it necessary, a nice to have or just a product if interest.
Many of the biggest technological hits, paradigm shifts or "the next big thing" have come from people working in their basement not knowing how things may turn out.
I laugh at news anchors trying to figure out the random video chat site at chatroulette. It is a big hit among young people. The only thing that the "older crowd" has to say is that there are lots of strange people out there and that it could be dangerous. The thing is for the older generation is that whatever they don't understand is usually labled as dangerous by them.
Large corporations and society faces the same problem, in that it is run by old people with old ideas. I'll give credit where it is due: in that they know how things worked and how to keep the status quo. These people are set in their ways and are moving in a certain direction.
The nice thing about being young is that we don't have any preconceptions about how things should be. We just don't know and when it comes to doing something, we generally end up trying to reinvent the wheel and sometimes we walk away with a completely new kind of wheel during the process. I think that is the power of youth that should be cherished instead of beaten out by our "societal system."
I found a musical duo that I really enjoy just because of it's "made at home" yet professional feel. And old piano and anything they have on them to turn into musical instruments. If you have time, check them out and their other songs. I love this kind of youth. More power to them.
As with moving bodies, newton's first law also seems to apply to life in motion; to those that are active, to those in inaction, or to those that need a change in vector.
One thing that I miss is being around creative technologists- people that like to do interesting things with technology. For the work environment I exist in, I just haven't been seeing the level of creativity that I want to see. Maybe it's the maturity level of the project that I've been working on but still, I haven't been able to take part to produce something with some level of personal creative satisfaction.
Some of the most fun projects I've worked on were simple things on my off hours with friends or on my own. More of like a dabbler but I would like to dedicate more time to learning how to do certain things with a professional flare maybe sometimes in the future. But usually, most of the best ideas I've seen have come out from people working in their garage to scratch some kind of itch, whether be it necessary, a nice to have or just a product if interest.
Many of the biggest technological hits, paradigm shifts or "the next big thing" have come from people working in their basement not knowing how things may turn out.
I laugh at news anchors trying to figure out the random video chat site at chatroulette. It is a big hit among young people. The only thing that the "older crowd" has to say is that there are lots of strange people out there and that it could be dangerous. The thing is for the older generation is that whatever they don't understand is usually labled as dangerous by them.
Large corporations and society faces the same problem, in that it is run by old people with old ideas. I'll give credit where it is due: in that they know how things worked and how to keep the status quo. These people are set in their ways and are moving in a certain direction.
The nice thing about being young is that we don't have any preconceptions about how things should be. We just don't know and when it comes to doing something, we generally end up trying to reinvent the wheel and sometimes we walk away with a completely new kind of wheel during the process. I think that is the power of youth that should be cherished instead of beaten out by our "societal system."
I found a musical duo that I really enjoy just because of it's "made at home" yet professional feel. And old piano and anything they have on them to turn into musical instruments. If you have time, check them out and their other songs. I love this kind of youth. More power to them.
Sunday, March 21, 2010
If Japanese thinking could meet Western thinking in the right way
Maybe I was fortunate to have had an interesting group of friends while I lived back in Canada. The kinds of people I knew are pretty much the kind of "make it on their own" people, self starters, in other words.
It doesn't just have to be in business or in the world of money making, but the skills that one does learn in doing things and starting thing on their own, most definitely apply here. And the unfortunate thing over here is that I haven't seen too many of those kinds of people in my age group over here. There are the few, but there aren't many.
I don't know if the generalization is true, but in most hierarchal societies, especially asian ones, most things are handed down from the elders to the younger generations. It's called seniority, I guess. And when people are stuck waiting around for something to be passed along to them, I don't think that many paradigm breaking ideas happen in a system like this. You need the idea and maybe go against mainstream conventional thinking, though I must also add that many controversial/revolutionary ideas were born in the same plight.
That isn't to say that most Asian people are like that. I have met many, especially those that are in music, art or some thing other than working your standard run of the mill company that are exactly like the 2 guys in a garage kind of thing. I just don't think I see or feel it in the technology department. These guys are really good at paying attention to details, even really minute ones that your ordinary westerner might pick up on.
But when it comes to systemic improvement, Japan is pretty hard to beat... Germany might be pretty close up there too from the ones that I know too.
If western "open ended" (ie. try crazy but cool things) thinking met the Japanese "improvement" way of thinking. You'd have a pretty hard combination to beat.
Finding the right people and the idea to make the plunge, however it tough, however.
It doesn't just have to be in business or in the world of money making, but the skills that one does learn in doing things and starting thing on their own, most definitely apply here. And the unfortunate thing over here is that I haven't seen too many of those kinds of people in my age group over here. There are the few, but there aren't many.
I don't know if the generalization is true, but in most hierarchal societies, especially asian ones, most things are handed down from the elders to the younger generations. It's called seniority, I guess. And when people are stuck waiting around for something to be passed along to them, I don't think that many paradigm breaking ideas happen in a system like this. You need the idea and maybe go against mainstream conventional thinking, though I must also add that many controversial/revolutionary ideas were born in the same plight.
That isn't to say that most Asian people are like that. I have met many, especially those that are in music, art or some thing other than working your standard run of the mill company that are exactly like the 2 guys in a garage kind of thing. I just don't think I see or feel it in the technology department. These guys are really good at paying attention to details, even really minute ones that your ordinary westerner might pick up on.
But when it comes to systemic improvement, Japan is pretty hard to beat... Germany might be pretty close up there too from the ones that I know too.
If western "open ended" (ie. try crazy but cool things) thinking met the Japanese "improvement" way of thinking. You'd have a pretty hard combination to beat.
Finding the right people and the idea to make the plunge, however it tough, however.
Wednesday, March 17, 2010
You don't have forever
If I had kids, this is probably one of the lessons I'd like to teach them, you don't have forever because between now and later, this might be the only chance you're going to get and there might be no going back.
Subconsciously, people do know what they want; ask them out right sometimes, it might not pop out, but given a chance it will come out. Anything that you've wished that you wanted to do, is something that you should be doing. If I were to summarize a huge number of life tragedies, it would be that.
Life works in time windows that you don't realize that they are there until they start closing. You generally can be reckless while young, it might even be expected, but don't expect to be able to do the same thing forever.
A big part of life is knowing how to age, gracefully. If there is something that you know that you ought to be doing, then do it until you don't. Because there are things that you can do now that you can't do later... so don't waste time.
And besides, people that make things happen or can do what they want is sexy.
Subconsciously, people do know what they want; ask them out right sometimes, it might not pop out, but given a chance it will come out. Anything that you've wished that you wanted to do, is something that you should be doing. If I were to summarize a huge number of life tragedies, it would be that.
Life works in time windows that you don't realize that they are there until they start closing. You generally can be reckless while young, it might even be expected, but don't expect to be able to do the same thing forever.
A big part of life is knowing how to age, gracefully. If there is something that you know that you ought to be doing, then do it until you don't. Because there are things that you can do now that you can't do later... so don't waste time.
And besides, people that make things happen or can do what they want is sexy.
Monday, March 01, 2010
What I miss about having roommates
It's been nearly 2 years since I've moved out to the Tokyo area and one of the things I realized that I am starting to miss is having roommates to chat with. What I noticed is the ability to speak well is as good as your ability to keep it sharp; if you stop, it will fade.
Since moving to Tokyo, I am immersed in the Japanese psyche; gone are the days of having international and bilingual Japanese friends. I could switch into English mode whenever I wanted and through that, I've kept my speaking, writing and thinking skills sharp. Having bilingual Japanese friends was also a big help since we could switch back between languages if things got tricky. Though I am pretty proficient at saying what I need to say in Japanese I consider myself far from eloquent in expression. In other words, don't expect any extraordinary oratories from me in Japanese (yet).
I believe that a person's linguistic skills are related to their ability to describe something logically, the succinctness of communication and the detail at which their thoughts are expressed. As it maybe subtle, many important thoughts and interesting conversations are usually the result of good attention to details, especially interesting details, otherwise communication gets boring (as to what would be considered "interesting" is a whole different problem all together, but I will assume that you know what I mean).
If you were to ask a person a simple question, "what do you do on the weekends?" The boring answer would be something like "oh, stuff" or "hang out with friends," which is dull for an answer. I used to chide students learning English conversation from me, "unless you don't like the person you're talking to, you're going to have to give a better response." The more interesting answer would be "I like the out doors and usually go cycling on sunny days to enjoy the good weather," which gives room for the conversation to go somewhere instead of the previous dead-end answer.
Being a good conversationalist depends on avoiding dead-end responses and finding other directions to take the conversation after reaching the end of a train of thought. Good comprehension and cognitive ability to process and respond to information on the fly is also required to hone a skill such as this. However sadly, there is no silver-bullet to getting that silver-tongue. Becoming a good conversationalist is hard and takes a considerable amount of time to be practiced in a second language because it requires a multitude of different skills. On the right topic and the correct frame of mind, I can usually hold a good conversation in Japanese, but if things head into "foreign territory" when I don't have the vocabulary or the understanding to respond appropriately, it gets tough.
The hardest environments to hold a conversation for me is with a medium to large group of Japanese people. Colloquial phrases and cultural references are usually hard to pick up and when the conversation gets exciting, the pace picks up making it harder to follow. It would be like expecting a foreigner to understand what some one means when "they got one upped" and someone else responds "don't worry, you'll take them down." And a dictionary isn't going to help either.
But if you really want to set the bar high and get in with the locals and have an impressive conversation, either through slang or verbal eloquence, it can be quite frustrating not being able to express complicated and detailed ideas in the manner that you are able to do so in your native tongue, even more so if you are very proficient. There has been many instances where I have faced this problem, to at some point where I've given up in some occasions because it would take too much effort. The deciding point is usually between the time it takes for me to explain something to someone with complete satisfaction vs the time I would lose to do something productive.
When one starts to give up on talking because it is tough, it becomes a self-fulling prophecy: the less you talk the harder it becomes to explain something so the the less you talk and so on. I even find myself losing my English proficiency without people to talk to on a regular basis. The prospect of that is freighting so I've decided to try and keep up with my English through writing and thought, this is the biggest thing I've miss about having roommates; having good communication.
Since moving to Tokyo, I am immersed in the Japanese psyche; gone are the days of having international and bilingual Japanese friends. I could switch into English mode whenever I wanted and through that, I've kept my speaking, writing and thinking skills sharp. Having bilingual Japanese friends was also a big help since we could switch back between languages if things got tricky. Though I am pretty proficient at saying what I need to say in Japanese I consider myself far from eloquent in expression. In other words, don't expect any extraordinary oratories from me in Japanese (yet).
I believe that a person's linguistic skills are related to their ability to describe something logically, the succinctness of communication and the detail at which their thoughts are expressed. As it maybe subtle, many important thoughts and interesting conversations are usually the result of good attention to details, especially interesting details, otherwise communication gets boring (as to what would be considered "interesting" is a whole different problem all together, but I will assume that you know what I mean).
If you were to ask a person a simple question, "what do you do on the weekends?" The boring answer would be something like "oh, stuff" or "hang out with friends," which is dull for an answer. I used to chide students learning English conversation from me, "unless you don't like the person you're talking to, you're going to have to give a better response." The more interesting answer would be "I like the out doors and usually go cycling on sunny days to enjoy the good weather," which gives room for the conversation to go somewhere instead of the previous dead-end answer.
Being a good conversationalist depends on avoiding dead-end responses and finding other directions to take the conversation after reaching the end of a train of thought. Good comprehension and cognitive ability to process and respond to information on the fly is also required to hone a skill such as this. However sadly, there is no silver-bullet to getting that silver-tongue. Becoming a good conversationalist is hard and takes a considerable amount of time to be practiced in a second language because it requires a multitude of different skills. On the right topic and the correct frame of mind, I can usually hold a good conversation in Japanese, but if things head into "foreign territory" when I don't have the vocabulary or the understanding to respond appropriately, it gets tough.
The hardest environments to hold a conversation for me is with a medium to large group of Japanese people. Colloquial phrases and cultural references are usually hard to pick up and when the conversation gets exciting, the pace picks up making it harder to follow. It would be like expecting a foreigner to understand what some one means when "they got one upped" and someone else responds "don't worry, you'll take them down." And a dictionary isn't going to help either.
But if you really want to set the bar high and get in with the locals and have an impressive conversation, either through slang or verbal eloquence, it can be quite frustrating not being able to express complicated and detailed ideas in the manner that you are able to do so in your native tongue, even more so if you are very proficient. There has been many instances where I have faced this problem, to at some point where I've given up in some occasions because it would take too much effort. The deciding point is usually between the time it takes for me to explain something to someone with complete satisfaction vs the time I would lose to do something productive.
When one starts to give up on talking because it is tough, it becomes a self-fulling prophecy: the less you talk the harder it becomes to explain something so the the less you talk and so on. I even find myself losing my English proficiency without people to talk to on a regular basis. The prospect of that is freighting so I've decided to try and keep up with my English through writing and thought, this is the biggest thing I've miss about having roommates; having good communication.
Sunday, February 28, 2010
Mascarpone meat balls
The nice thing about using a bicycle is that it is easier to part in the city compared to a scooter and I've been visiting the nicer grocery stores which are a little ways off from where I live to pick up some better (and cheaper!) ingredients for cooking.
I found a great recipe for meatballs using ricotta cheese. Unfortunately I couldn't find ricotta cheese and substituted it for marscapone cheese instead. The recipe calls for ground beef, parsley (and plenty of it), diced and slightly fried onions with garlic, bread crumbs, an egg and salt and pepper to taste.
The meat balls made at first doesn't look like the most appetizing thing in the world as they look very gooey at first, however the point of the cheese is to keep the meatballs damp and moist after cooking. I can tell you that the cheese did an excellent job of doing that.
After frying the meat balls in olive oil and soaking out the oil with tissue paper, I dumped in 2 chopped tomatoes and added a little tomato sauce for some extra kick which led to the following.
Now those look like great meatballs. The cooking process took about 20~30 minutes and good care is required when flipping these meat balls at first as they seemed a little easy to break. But after being cooked through thoroughly, they hold together quite nicely. After that, I just had to put them over some spaghetti and I was good for dinner.
The image was a little blurry since I was more interested in eating dinner than taking photos. But those were the most juicy meatballs I've ever had. Something I'll make time to do again.
I found a great recipe for meatballs using ricotta cheese. Unfortunately I couldn't find ricotta cheese and substituted it for marscapone cheese instead. The recipe calls for ground beef, parsley (and plenty of it), diced and slightly fried onions with garlic, bread crumbs, an egg and salt and pepper to taste.
The meat balls made at first doesn't look like the most appetizing thing in the world as they look very gooey at first, however the point of the cheese is to keep the meatballs damp and moist after cooking. I can tell you that the cheese did an excellent job of doing that.
After frying the meat balls in olive oil and soaking out the oil with tissue paper, I dumped in 2 chopped tomatoes and added a little tomato sauce for some extra kick which led to the following.
Now those look like great meatballs. The cooking process took about 20~30 minutes and good care is required when flipping these meat balls at first as they seemed a little easy to break. But after being cooked through thoroughly, they hold together quite nicely. After that, I just had to put them over some spaghetti and I was good for dinner.
The image was a little blurry since I was more interested in eating dinner than taking photos. But those were the most juicy meatballs I've ever had. Something I'll make time to do again.
My new cross bike
It's winter and in an effort to get into more shape, I've decided to buy a new bicycle --> a cross bike, which is a hybrid between a road bike and a mountain bike. The easiest way to describe the bike is a road bike (the ones with really thin wheels) with slightly thicker wheels.
I did some shopping around at various stores near by where lived and most of these bikes go for about $500~$700 for a cheap cross bike, but with the advent of the internet, I managed to find one going for about $400 which was a good deal in my books.
The bike I bought is a 11.5 kg, 24 speed, aluminum body cross bike. The thing arrived last month and I am loving the things to bits. The last time I had a bike with any sort of real gearing was when I was in Canada and I am quite certain that the old bike I had was easily in the 20 kg range. The bike I got looks like this:
The new cross bike
This puts out an incredible amount of speed with barely any input in pedaling power. I estimate that I can easily go 30 km/h and pushing it a little, up to about 40 km/h. I am certain that I am a thorn in most driver's sides as I am a pain in the rear to pass in single lane streets-- just going fast enough that it takes more than a few second window to pass me that it's find to find that opening in the on coming lane. As much as I might be a pain for the drivers, I really enjoy riding this bike.
I really never was that interested in long distance cycling before but after taking this bike out for a spin after work and on sunny weekends, I can easily see my self going on 30 km+ adventures. It is much easier to make the decision to stop somewhere and check things out on a bike than compared to driving in a car or riding my scooter; a car being big to park on the side of the road and a scooter goes fast enough making it hard to make the spit second decision to stop somewhere (I usually end up doubling back if I see something interesting). Riding a bike around town is great for photo shoot adventures and below are just a few shots of what I've taken on a ride I did in early February:
A bridge under construction
A rice field after harvest
Though I've have a scooter, this bike gives me new reason to go out exploring again and I am having a great time doing it. My next long ride will be down to Kamakura, which is a wonderful stretch of beach about 15 km from where I live. Plenty of things to see and eat there so I'll be looking forward to touring around during the next sunny weekend. If that goes well, I am looking to trying a weekend tour where I'll push about 150~200 km to go from where I live to Tokyo and then come back home.
For long tours, I am tempted at buying a GPS and playing around with geo-tagging photos, where my photos are automatically tagged to where they were taken. It would be awesome to keep track of all the interesting places I've visited. My friends have recommend that I get an iPhone since it had a built in GPS. It's a very tempting proposition and the costs in Japan are fairly good. Many people use the iPhone for data only and pay about $25/month for it so it's something I'll have to look into.
The goal is to slim down and remove some excess fat for the summer, we'll see how this goes!
I did some shopping around at various stores near by where lived and most of these bikes go for about $500~$700 for a cheap cross bike, but with the advent of the internet, I managed to find one going for about $400 which was a good deal in my books.
The bike I bought is a 11.5 kg, 24 speed, aluminum body cross bike. The thing arrived last month and I am loving the things to bits. The last time I had a bike with any sort of real gearing was when I was in Canada and I am quite certain that the old bike I had was easily in the 20 kg range. The bike I got looks like this:
The new cross bikeThis puts out an incredible amount of speed with barely any input in pedaling power. I estimate that I can easily go 30 km/h and pushing it a little, up to about 40 km/h. I am certain that I am a thorn in most driver's sides as I am a pain in the rear to pass in single lane streets-- just going fast enough that it takes more than a few second window to pass me that it's find to find that opening in the on coming lane. As much as I might be a pain for the drivers, I really enjoy riding this bike.
I really never was that interested in long distance cycling before but after taking this bike out for a spin after work and on sunny weekends, I can easily see my self going on 30 km+ adventures. It is much easier to make the decision to stop somewhere and check things out on a bike than compared to driving in a car or riding my scooter; a car being big to park on the side of the road and a scooter goes fast enough making it hard to make the spit second decision to stop somewhere (I usually end up doubling back if I see something interesting). Riding a bike around town is great for photo shoot adventures and below are just a few shots of what I've taken on a ride I did in early February:
A bridge under constructionA rice field after harvestFor long tours, I am tempted at buying a GPS and playing around with geo-tagging photos, where my photos are automatically tagged to where they were taken. It would be awesome to keep track of all the interesting places I've visited. My friends have recommend that I get an iPhone since it had a built in GPS. It's a very tempting proposition and the costs in Japan are fairly good. Many people use the iPhone for data only and pay about $25/month for it so it's something I'll have to look into.
The goal is to slim down and remove some excess fat for the summer, we'll see how this goes!
Sunday, February 21, 2010
More writing in a bit...
Bought a new cross bike recently and have been riding it to bits lately.
Been quite an active weekend and will have photos up in a bit. It's hard finding time to relax, write, go out and do things and juggle work all at the same time.
I read in a book once, that once you start working, life becomes a process of making sacrifices. As much as I hate the thought of it, it seems kind of true.
Been quite an active weekend and will have photos up in a bit. It's hard finding time to relax, write, go out and do things and juggle work all at the same time.
I read in a book once, that once you start working, life becomes a process of making sacrifices. As much as I hate the thought of it, it seems kind of true.
Friday, February 12, 2010
Visualizing the Stock Market
I've finally gotten the stock market analysis program up to a point where I can start plotting data again. There has been some issues with robustness of the program vs the data I am using so I've had to lots of debugging and internal checks to get the code base where it is now. I am confident for the time being that I should be able to start ramping up on writing code to analyze general stock market data to understand the stock market better.
This project has been nearly 2~3 years since starting as a pet project while I was in university. The code base was originally developed in Matlab, but had many limitations when it came to running parallel processes and speed issues. After nearly 3~4 rewrites of the code base, I have finally settled down on the data structures and how to interact with the data. There is a saying when it comes to programming in that you should be prepared to throw out the first few versions of your software.
The program I am developing at the moment is mostly educational, to help me look at the stock market in different ways compared to what is presented at most financial webpages. Looking at the stock market through the S&P 500 and the Dow Jones Industrial index gives a very limited look at the entire stock market and there is a lot of information lost when everything is averaged together. Why look at only 100 to 500 stocks when you can look at nearly 3400 all at once to have an idea of what is going on?
Though I have been participating in the stock market in the last 4~5 years, I never did feel that I understood the big picture of the stock market. The most basic question that I desire to understand is to find out the "right price" for a share in some company. I have read a few investment books and most of them give the general idea that companies with good growing net incomes, low debt and risk factors are good companies to invest in. I generally would have to agree, but what price would you pay for a share in that company? If the share was $20 would you buy it? What about if it was $50 instead? Would it still be a good buy?
The problem with the stock market is that there is no canonical analysis technique to determine the price of a stock. Canonical, being a procedural method when applied by any person result in the same answer. Thus there is no way to say "because condition A therefore the result is B." At best, we may only make predictions, in other words "because condition A result is probably B." Because of this mode of thought, I believe that it is helpful to take a statistical approach to looking at the stock market (though I am against technical trading). In this endeavor, it is necessary to look collect and analyze larger amounts of data an apply decision making strategies and determine how often you are "right" (where being "right is still ill defined at the moment...)
The first few phases of this project has finally been completed, where I have been finally able to get stock market data and plot them for the majority of the listed companies. I've made one plot, which is a Share Price vs EPS plot. One would generally expect that the share price is somewhat correlated to the EPS, which is something we see here with a very wide band going upwards and towards the right in this plot.
Share Price vs EPS
What we also notice from this plot is that there is a large scattering of points to the left side of the graph and with few points scattered to the right. Points to the far left represent stocks that are probably over valued and points to the right are stocks that are possibly undervalued. From what we see in the plot here is that there are many companies out there with a very negative EPS but still high stock price. What has yet to be determined is if the market is expecting those stocks to recover or if those stocks are truly over priced.
This project has been nearly 2~3 years since starting as a pet project while I was in university. The code base was originally developed in Matlab, but had many limitations when it came to running parallel processes and speed issues. After nearly 3~4 rewrites of the code base, I have finally settled down on the data structures and how to interact with the data. There is a saying when it comes to programming in that you should be prepared to throw out the first few versions of your software.
The program I am developing at the moment is mostly educational, to help me look at the stock market in different ways compared to what is presented at most financial webpages. Looking at the stock market through the S&P 500 and the Dow Jones Industrial index gives a very limited look at the entire stock market and there is a lot of information lost when everything is averaged together. Why look at only 100 to 500 stocks when you can look at nearly 3400 all at once to have an idea of what is going on?
Though I have been participating in the stock market in the last 4~5 years, I never did feel that I understood the big picture of the stock market. The most basic question that I desire to understand is to find out the "right price" for a share in some company. I have read a few investment books and most of them give the general idea that companies with good growing net incomes, low debt and risk factors are good companies to invest in. I generally would have to agree, but what price would you pay for a share in that company? If the share was $20 would you buy it? What about if it was $50 instead? Would it still be a good buy?
The problem with the stock market is that there is no canonical analysis technique to determine the price of a stock. Canonical, being a procedural method when applied by any person result in the same answer. Thus there is no way to say "because condition A therefore the result is B." At best, we may only make predictions, in other words "because condition A result is probably B." Because of this mode of thought, I believe that it is helpful to take a statistical approach to looking at the stock market (though I am against technical trading). In this endeavor, it is necessary to look collect and analyze larger amounts of data an apply decision making strategies and determine how often you are "right" (where being "right is still ill defined at the moment...)
The first few phases of this project has finally been completed, where I have been finally able to get stock market data and plot them for the majority of the listed companies. I've made one plot, which is a Share Price vs EPS plot. One would generally expect that the share price is somewhat correlated to the EPS, which is something we see here with a very wide band going upwards and towards the right in this plot.
Share Price vs EPSSunday, January 31, 2010
The art of seduction
I started reading a book today because I had something interesting happen to me yesterday.
I was at a party hosted by a friend (a wedding/singles after party) for people to mingle and I was there just to meet people. I do what that should come naturally, meet and introduce yourself to the people around you. So it turns out that I met a very pretty Singaporean babe that just recently moved to Japan. Very out doorsy and we had lots to talk about. The conversation was great and we kicked it off really well.
While at the bar, there were some other guys hovering about, I didn't realize it at the time but I was being sized up as a competitor buy some English guys eager for my friend. Over the course of the night, I realized that I had already missed the last train home and casually asked if I could stay over till the morning, with obviously no ulterior motives. I generally don't like to be a "player" as a personal preference. One of the odd things about me is that sometimes I don't realize that I was subconsciously seducing this person since the conversation was going pretty well and some other guys were wanting in on the action. How they played me off was pretty slick but they couldn't separate the girl from me through out the entire evening.
One guy ran cover to try and get me into a conversation with him while the other guy tried to work the girl elsewhere. I caught on near the end and the ironic thing is that they got so completely desperate that after getting the girl's phone number to try and invite her after the party out for drinks. The ending was that I got double teamed by these guys and they confronted me to "get lost" in a subliminal way. I wasn't interested in bedding the girl, though she would have been fun to hang out with. I wasn't looking for a fight and buggered off and after replaying the events through out the evening in my head and everything clicked and I realized my mistake, I should have put up a fight and told them to screw off.
I decided to start reading a book today, called "The art of seduction" since I had a feeling that I might find something interesting about today's event, an analysis of the characters I bumped into today. I've only gotten through the preface and found this gem:
"Finally, seducers are completely amoral in their approach to life. It is all
a game, an arena for play. Knowing that the moralists, the crabbed repressed
types who croak about the evils of the seducer, secretly envy their power,
they do not concern themselves with other people's opinions."
-The Art of Seduction
Immediately the image of the two guys that worked me for the girl came to mind. I plan on reading through this book because it's a fascinating read to help me understand the games these people play and counter them. Feels like politics but it's a game that I am going to have to know how to play.
I might have lost this time around, but it was an interesting eye opener. Also as an aside, the event I went was filled with people in the financial world, all sorts of bankers and etc. The one thing that I felt was that many of them had the "seducer" archetype embedded in them in some manner. Power hungry, commanding, persuasive. As long as you're their friend you're fine, but if you're on the wrong side, all bets are off. It's a cut-throat world.
I was at a party hosted by a friend (a wedding/singles after party) for people to mingle and I was there just to meet people. I do what that should come naturally, meet and introduce yourself to the people around you. So it turns out that I met a very pretty Singaporean babe that just recently moved to Japan. Very out doorsy and we had lots to talk about. The conversation was great and we kicked it off really well.
While at the bar, there were some other guys hovering about, I didn't realize it at the time but I was being sized up as a competitor buy some English guys eager for my friend. Over the course of the night, I realized that I had already missed the last train home and casually asked if I could stay over till the morning, with obviously no ulterior motives. I generally don't like to be a "player" as a personal preference. One of the odd things about me is that sometimes I don't realize that I was subconsciously seducing this person since the conversation was going pretty well and some other guys were wanting in on the action. How they played me off was pretty slick but they couldn't separate the girl from me through out the entire evening.
One guy ran cover to try and get me into a conversation with him while the other guy tried to work the girl elsewhere. I caught on near the end and the ironic thing is that they got so completely desperate that after getting the girl's phone number to try and invite her after the party out for drinks. The ending was that I got double teamed by these guys and they confronted me to "get lost" in a subliminal way. I wasn't interested in bedding the girl, though she would have been fun to hang out with. I wasn't looking for a fight and buggered off and after replaying the events through out the evening in my head and everything clicked and I realized my mistake, I should have put up a fight and told them to screw off.
I decided to start reading a book today, called "The art of seduction" since I had a feeling that I might find something interesting about today's event, an analysis of the characters I bumped into today. I've only gotten through the preface and found this gem:
"Finally, seducers are completely amoral in their approach to life. It is all
a game, an arena for play. Knowing that the moralists, the crabbed repressed
types who croak about the evils of the seducer, secretly envy their power,
they do not concern themselves with other people's opinions."
-The Art of Seduction
Immediately the image of the two guys that worked me for the girl came to mind. I plan on reading through this book because it's a fascinating read to help me understand the games these people play and counter them. Feels like politics but it's a game that I am going to have to know how to play.
I might have lost this time around, but it was an interesting eye opener. Also as an aside, the event I went was filled with people in the financial world, all sorts of bankers and etc. The one thing that I felt was that many of them had the "seducer" archetype embedded in them in some manner. Power hungry, commanding, persuasive. As long as you're their friend you're fine, but if you're on the wrong side, all bets are off. It's a cut-throat world.
Wednesday, January 27, 2010
Being paid by the hour = work stays at work
I am paid pretty much by the hour, some fixed rate during the day and hourly for overtime. The one thing I realized is that I've developed a "if I'm not going to be paid for it, then I am not going to do it" type attitude. Meaning that I don't generally take work home to do during my off hours.
It's an interesting contrast compared to my university days when there was no such divide; the objective was to get the best possible grades in class and thus I did what I had to get good grades. It would be like the difference between getting grades for "going to class" versus "doing well on a test." The difference being similar to a salary vs performance based pay.
My time is as such, allocated in such a manner. If I have work to do, I'll be sure to try and to it at work, even though it might be nice to work from the comfort of home from time to time. When I am home and want to study something like the stock market and figure out how to profit on it, then I'd gladly spend my evenings doing research.
I guess, it's about following the incentive, but in this case, it seems to be that the room for personal development is more obvious compared to working a salary based job.
It's an interesting contrast compared to my university days when there was no such divide; the objective was to get the best possible grades in class and thus I did what I had to get good grades. It would be like the difference between getting grades for "going to class" versus "doing well on a test." The difference being similar to a salary vs performance based pay.
My time is as such, allocated in such a manner. If I have work to do, I'll be sure to try and to it at work, even though it might be nice to work from the comfort of home from time to time. When I am home and want to study something like the stock market and figure out how to profit on it, then I'd gladly spend my evenings doing research.
I guess, it's about following the incentive, but in this case, it seems to be that the room for personal development is more obvious compared to working a salary based job.
Sunday, January 24, 2010
Having a good software design document is important
I've come to the point in my software project where it's starting to get hard to manage making tweaks to my code and I am starting to realize that it would have been nice to have a software design document to keep track of what is going on in the code. I'm getting to the point where making a tweak in one part of my code requires tweaking of other bits of code to make everything work together properly is starting to consume a lot of time. This is one reason why sometimes rewriting things from the ground up can be faster than incrementally modifying a program.
Writing a program is like writing an essay. In the case of the essay, you are writing to figure out what you want to say. Sometimes after many paragraphs, thesis', topic sentences and self reflecting discussions later, the initial premise of the essay could be completely or partially wrong, resulting in a rewrite of various sections. This is especially so for long or complicated essays and often having a outline of what you want to write can be a huge time saver-- instead of writing 2000 words to and then realizing that the logic was bad, you might have been able to write an outline in 200 words and figured that out earlier. It also helps when it comes to keeping track of what needs to be said.
In the case of a program, you are writing a set of procedures on what to do with data. So instead of having a thesis, topic sentences and discussions you are describing sets of data and the the kinds of operations that must be performed on the data. The difference between the essay and writing a program is the difference between figuring out what you want to say versus what you want to do.
Programming is an organic process
Like writing and essay to figure something out, writing a program is about figuring out what you want to do and how you're going to do it. For my current project with parsing html data from financial webpages and then performing analysis, I didn't have debugging in mind and decided that it would be OK to download data from the web, parse the HTML in memory, save the values in a table and then save it when I was done processing everything. I ran into significant problems when the program choked on bad html data and crashed. Since I wasn't saving the webpage in question, I wasn't able to easily recreate the error and I would have to manually download the webpage and then perform an analysis on my own. After that, I would restart the process and hope that I wouldn't run into another problem (which I usually did!).
What I learned from that experience was that it was important to keep a hard copy of the raw data and write a parsing module that would be able to look at bad data, write a report and continue onwards. When I first designed my program, I expected all available data to be properly formatted, which unfortunately wasn't true. The result of this ordeal resulted in me rewriting my download module from processing data on the fly (and thus saving HD space) to saving the data first and then separating the parsing module and have it access and read the HTML file from the hard drive instead.
Other problems also started to creep up, I was also incorporating data from both yahoo and google and I assumed that I would be able to get complete data from both financial sites. Unfortunately, that wasn't the case and there were many times when my analysis module would break down because I was missing data. I spent plenty of time debating what was the best solution to deal with this problem. Do I have the analysis module detect the missing data? or do I go back into the parsing module and add functionality to prevent data sets with missing data from getting put into the database? While writing code for the analysis module, it's tempting to just add some checking functionality there since it would break my train of thought... I used to build in functionality where ever I needed it without concern for organization. This would balloon into a rat's nest of functions scattered all over in the code that became problematic in figuring out what piece of code was doing what.
Walking the line between a creative and structured process
Writing code is both a creative and structured process, a point that I didn't appreciate until now. There is the design and organization aspect where one needs to decide the kinds of modules and the input-output functionality required for each module which would fall under the creative process. The structured process would be taking the outlined requirements and then translating that in to robust and tested code. It sounds easy when things are properly broken down, but it can turn into a hairy mess when one starts to realize that there were some missing things in the initial design. Without a design document, it becomes tempting to start making design decisions on the fly to get things done or if one start thinking too much it becomes pain to try and visualize the whole program and figure out how things might affect one or another part of the code.
I've been in both situations where I would an hour trying to visualize the entire program and trying to make a decision on what to do. After finally making a decision, the implementation might take only 20 minutes and I'd already be exhausted from all the thinking and still not be confident that I made the right decision. Running into road blocks like these can be a really frustrating experience.
All of this can be avoided by keeping a well maintained design document that describes the functionality of the program in it's current state where it can be used as a reference for further design revisions. Writing a design document, though, doesn't particularly feel like really coding and it is tempting to skip over it, but I realize now that keeping a decent outline of code and it's status is really helpful. I'll probably be spending my next few slots of free time creating a design document.
Writing a program is like writing an essay. In the case of the essay, you are writing to figure out what you want to say. Sometimes after many paragraphs, thesis', topic sentences and self reflecting discussions later, the initial premise of the essay could be completely or partially wrong, resulting in a rewrite of various sections. This is especially so for long or complicated essays and often having a outline of what you want to write can be a huge time saver-- instead of writing 2000 words to and then realizing that the logic was bad, you might have been able to write an outline in 200 words and figured that out earlier. It also helps when it comes to keeping track of what needs to be said.
In the case of a program, you are writing a set of procedures on what to do with data. So instead of having a thesis, topic sentences and discussions you are describing sets of data and the the kinds of operations that must be performed on the data. The difference between the essay and writing a program is the difference between figuring out what you want to say versus what you want to do.
Programming is an organic process
Like writing and essay to figure something out, writing a program is about figuring out what you want to do and how you're going to do it. For my current project with parsing html data from financial webpages and then performing analysis, I didn't have debugging in mind and decided that it would be OK to download data from the web, parse the HTML in memory, save the values in a table and then save it when I was done processing everything. I ran into significant problems when the program choked on bad html data and crashed. Since I wasn't saving the webpage in question, I wasn't able to easily recreate the error and I would have to manually download the webpage and then perform an analysis on my own. After that, I would restart the process and hope that I wouldn't run into another problem (which I usually did!).
What I learned from that experience was that it was important to keep a hard copy of the raw data and write a parsing module that would be able to look at bad data, write a report and continue onwards. When I first designed my program, I expected all available data to be properly formatted, which unfortunately wasn't true. The result of this ordeal resulted in me rewriting my download module from processing data on the fly (and thus saving HD space) to saving the data first and then separating the parsing module and have it access and read the HTML file from the hard drive instead.
Other problems also started to creep up, I was also incorporating data from both yahoo and google and I assumed that I would be able to get complete data from both financial sites. Unfortunately, that wasn't the case and there were many times when my analysis module would break down because I was missing data. I spent plenty of time debating what was the best solution to deal with this problem. Do I have the analysis module detect the missing data? or do I go back into the parsing module and add functionality to prevent data sets with missing data from getting put into the database? While writing code for the analysis module, it's tempting to just add some checking functionality there since it would break my train of thought... I used to build in functionality where ever I needed it without concern for organization. This would balloon into a rat's nest of functions scattered all over in the code that became problematic in figuring out what piece of code was doing what.
Walking the line between a creative and structured process
Writing code is both a creative and structured process, a point that I didn't appreciate until now. There is the design and organization aspect where one needs to decide the kinds of modules and the input-output functionality required for each module which would fall under the creative process. The structured process would be taking the outlined requirements and then translating that in to robust and tested code. It sounds easy when things are properly broken down, but it can turn into a hairy mess when one starts to realize that there were some missing things in the initial design. Without a design document, it becomes tempting to start making design decisions on the fly to get things done or if one start thinking too much it becomes pain to try and visualize the whole program and figure out how things might affect one or another part of the code.
I've been in both situations where I would an hour trying to visualize the entire program and trying to make a decision on what to do. After finally making a decision, the implementation might take only 20 minutes and I'd already be exhausted from all the thinking and still not be confident that I made the right decision. Running into road blocks like these can be a really frustrating experience.
All of this can be avoided by keeping a well maintained design document that describes the functionality of the program in it's current state where it can be used as a reference for further design revisions. Writing a design document, though, doesn't particularly feel like really coding and it is tempting to skip over it, but I realize now that keeping a decent outline of code and it's status is really helpful. I'll probably be spending my next few slots of free time creating a design document.
Saturday, January 23, 2010
You're lucky if it's "garbage in, garbage out"
When it comes to code, you don't usually have control over the data coming in. Usually I write code that I use on my own, when I know what kind of input to expect from myself or other parts of the program. Things don't break quite as easily. But when it comes to dealing with information out of the web, especially parsing data out of an html file, you don't have that luxury. Things can break on a moment's notice when someone decides to change the data format or communication errors resulting in bad data.
I can have some primitive code up and running pretty quick but the program will crash at the first sign of bad input. If you're analyzing large data sets, the last thing that you want is the code crashing when you have bad data. Graceful error handling is important and also a pain in the ass to deal with.
Firstly there are a whole slew of potential errors that could occur that are hard to predict. Download connections could be reset, the available data could be incomplete, there could be an error in the input format. Then if you are cross referencing data from other sources, you have to make sure that the data you want to look at for all sources is available and who knows how things might break in the future.
Having robust code that can detect errors and report back can easily double the effort required to write a program. It's hard to realize what needs to be made more robust until things start breaking. I had a basic version of a stock analysis program working last year but I was continually hacking parts of the program to ignore bad data. I generally resorted to "execute and pray" kind of tactic to see if I would get any results.
And so error handling and data integrity verification has become another important pillar to good programming that I've just realized and will have to spend more time devising a good strategy on dealing with this kind of stuff.
I can have some primitive code up and running pretty quick but the program will crash at the first sign of bad input. If you're analyzing large data sets, the last thing that you want is the code crashing when you have bad data. Graceful error handling is important and also a pain in the ass to deal with.
Firstly there are a whole slew of potential errors that could occur that are hard to predict. Download connections could be reset, the available data could be incomplete, there could be an error in the input format. Then if you are cross referencing data from other sources, you have to make sure that the data you want to look at for all sources is available and who knows how things might break in the future.
Having robust code that can detect errors and report back can easily double the effort required to write a program. It's hard to realize what needs to be made more robust until things start breaking. I had a basic version of a stock analysis program working last year but I was continually hacking parts of the program to ignore bad data. I generally resorted to "execute and pray" kind of tactic to see if I would get any results.
And so error handling and data integrity verification has become another important pillar to good programming that I've just realized and will have to spend more time devising a good strategy on dealing with this kind of stuff.
Friday, January 22, 2010
Positive pressure
I came home early tonight, 7 being relatively early for Japan.
My work schedule last year in was just damn busy and I honestly needed a break. I had 2 weeks of during the winter and I cane back to work on the 5th of this month and I still don't really feel like working that hard. Nope, none of those 4 hours over time everyday kind of schedule. I just want to see the clock hit 5:30 and work a little bit more and then head home to rest.
And I have been in this resting mode for the last few weeks. Nothing particularly productive being done, but just doing some occasional studying, programming and watching a hell of a lot of The West Wing, which is a series about really busy people running the White House, which I find is really ironic for a guy like me.
Right now, I don't have any real motivation at the moment. The work just doesn't stop flowing and there is alwys some issue that needs to be dealt with; welcome to the world of R&D. As much as the end result of the project is going to be sexy, the process it self is very far from it, and I hate boring work. I wonder what would be exciting work... then again I might be afraid of the answer.
So with the work being a little on the boring side, I have realized one thing is that I've become complacent with my way of life and that way of life is just "getting by" as in finishing work, coming home, taking a breather and repeating the process till the weekend until I use that to relax.
This life style has turned into a habit, a habit that I am finding hard to break out of. I know that I should be doing something new or at least progressing on some kind of goal, but it's oh so easy to just work, come home, rest and repeat. Creatures of habit as some people might consider us.
So I've looked for change, something to help me break out of this habit and to do something I feel excited about. It's strange to realize that I haven't felt excited about doing something in a long time, not since going backpacking through Asia or another time that I thought it would be so cool to make an equalizer circuit for a presentation on Lapalce transforms for a seminar.
I miss feeling excited about doing something and being with people that I can be excited with. Not just to do something crazy like a dare but excited in a productive way, the desire to push my self beyond boundaries to do something new. So right now, I am looking for something to be excited about, some positive pressure in life to do something enjoyable. Gotta find a way to break the habit.
My work schedule last year in was just damn busy and I honestly needed a break. I had 2 weeks of during the winter and I cane back to work on the 5th of this month and I still don't really feel like working that hard. Nope, none of those 4 hours over time everyday kind of schedule. I just want to see the clock hit 5:30 and work a little bit more and then head home to rest.
And I have been in this resting mode for the last few weeks. Nothing particularly productive being done, but just doing some occasional studying, programming and watching a hell of a lot of The West Wing, which is a series about really busy people running the White House, which I find is really ironic for a guy like me.
Right now, I don't have any real motivation at the moment. The work just doesn't stop flowing and there is alwys some issue that needs to be dealt with; welcome to the world of R&D. As much as the end result of the project is going to be sexy, the process it self is very far from it, and I hate boring work. I wonder what would be exciting work... then again I might be afraid of the answer.
So with the work being a little on the boring side, I have realized one thing is that I've become complacent with my way of life and that way of life is just "getting by" as in finishing work, coming home, taking a breather and repeating the process till the weekend until I use that to relax.
This life style has turned into a habit, a habit that I am finding hard to break out of. I know that I should be doing something new or at least progressing on some kind of goal, but it's oh so easy to just work, come home, rest and repeat. Creatures of habit as some people might consider us.
So I've looked for change, something to help me break out of this habit and to do something I feel excited about. It's strange to realize that I haven't felt excited about doing something in a long time, not since going backpacking through Asia or another time that I thought it would be so cool to make an equalizer circuit for a presentation on Lapalce transforms for a seminar.
I miss feeling excited about doing something and being with people that I can be excited with. Not just to do something crazy like a dare but excited in a productive way, the desire to push my self beyond boundaries to do something new. So right now, I am looking for something to be excited about, some positive pressure in life to do something enjoyable. Gotta find a way to break the habit.
Monday, January 18, 2010
Insufficient data vs a bad model
I nearly blew a gasket trying to explain to a co-worker the difference between a calculation model being incomplete and having insufficient data. I was trying to do something a little tricky which these guys have never done, calculate the fundamental parameters of a diode using measured current-voltage data.
The problem I was trying to solve was backwards from the problems that most text books give you. The text books give you the diode parameters and maybe a resistor in the circuit and you had to go and calcualate the current flowing through the circuit at some voltage.
I managed to setup the basic diode resistor equation and then startted back solving for the unknown coefficients. For those that are interested the equation is:
V = I*R + n*Vth*Ln(I/I_s + 1)
with the unknown factors being n, Is and R and you are given a set of IV data with some noise.
This problem is impossible to solve algebraically so one does need to use a numerical solver to find the most optimal values of n, Is and R to fit some modeled data to the curve. Obviously there are likely other effects that aren't in the model, such as temperature effects which may cause the resistance to change. I made the simplyifying assumption that the temperature change was insignificant enough to ignore and proceeded to do my calculations through a program I wrote.
The results I got was interesting, that I realized that we were taking insufficient data to for my simulator to make an accurate fit. For those that know diodes, diodes are non-linear devices, where the amount of current that goes through them is exponential in voltage or I = Is*e(V/Vth), resistors on the other hand, are linear devices and the current that goes through resistors is proportional to the voltage or I = I/R.
The interesting thing about having a diode and resistor in series is that the diode starts off as being the current limiting device at really low voltages (because the exponential is really small at first) then the resistor becomes the current limiting component at higher voltages. The problem was that our measurements had the current and voltage too low for us me to calculate a reasonable value for resistance, since resistance measurements becomes far more accurate because the resistor is influencing the circuit the most. For example, I modeled out a 1300 ohm resistance with a limited data set and then a 200 ohm resistance with an expanded data set.
Immediately after, everyone thought that the model was wrong since a different resistance was coming out after using a data say from 0 to 600 uA versus 0 to 2 mA. The biggest argument was that the model was flawed because I was getting different current values using different ranges of data. My head wanted to explode because I was telling them and had verified that we were getting in sufficient data.
The simplest example to illustrate this would be like the following example. Suppose that you had the following function: y = -a*x + b*x^2 + c*x^4 where a, b and c are unknown. You can measure both (x,y) and your objective is to find (a,b,c). There is some measurement noise when you get y and y is > 0
Suppose that you took data with x between 0 and 1, -a*x term is going to be the strongest and you'll probably be able to rig the equation to be always positive for some value of (a,b,c) and look about right for x between 0 and 1. But what if you took the same parameters and measured beyond [0,1] and up to [0,100]? How good would your fit be then? if your c*x^4 term had a lot of error, then the resulting graph is going to be out of whack. The (a,b,c) are going to have to be rejiggled so that the fit works well for both [0,1] and [0,100], the whole range. The values might change drastically!
And that is the problem when people try to fit limited data with higher order polynomic or non-linear functions. You could have many different functions that could describe a line segment very well, but everything falls apart when the line segment is extended, because the higher order terms were not fitted well.
I tried to explain that with all my might in Japanese. But they just didn't get it and they kept going with that the model was probably missing a few things. I even back calculated and showed reversed approximations giving a confidence interval down to 12% on the data, but alas no dice.
This is quite a problem when most people I work with are experimentalists, they just do stuff to see if things work or not. When it comes to simulation, programming or theoretical stuff, they know the basics but the real fruits are hidden in the details. Since we make some of these devices, it is important that they are characterized properly so we might have a good idea on what factors might impact the performance of these devices. Just saying that voltage is up or down isn't sufficient because you want to go after the biggest impact factors when improving things first.
The problem I was trying to solve was backwards from the problems that most text books give you. The text books give you the diode parameters and maybe a resistor in the circuit and you had to go and calcualate the current flowing through the circuit at some voltage.
I managed to setup the basic diode resistor equation and then startted back solving for the unknown coefficients. For those that are interested the equation is:
V = I*R + n*Vth*Ln(I/I_s + 1)
with the unknown factors being n, Is and R and you are given a set of IV data with some noise.
This problem is impossible to solve algebraically so one does need to use a numerical solver to find the most optimal values of n, Is and R to fit some modeled data to the curve. Obviously there are likely other effects that aren't in the model, such as temperature effects which may cause the resistance to change. I made the simplyifying assumption that the temperature change was insignificant enough to ignore and proceeded to do my calculations through a program I wrote.
The results I got was interesting, that I realized that we were taking insufficient data to for my simulator to make an accurate fit. For those that know diodes, diodes are non-linear devices, where the amount of current that goes through them is exponential in voltage or I = Is*e(V/Vth), resistors on the other hand, are linear devices and the current that goes through resistors is proportional to the voltage or I = I/R.
The interesting thing about having a diode and resistor in series is that the diode starts off as being the current limiting device at really low voltages (because the exponential is really small at first) then the resistor becomes the current limiting component at higher voltages. The problem was that our measurements had the current and voltage too low for us me to calculate a reasonable value for resistance, since resistance measurements becomes far more accurate because the resistor is influencing the circuit the most. For example, I modeled out a 1300 ohm resistance with a limited data set and then a 200 ohm resistance with an expanded data set.
Immediately after, everyone thought that the model was wrong since a different resistance was coming out after using a data say from 0 to 600 uA versus 0 to 2 mA. The biggest argument was that the model was flawed because I was getting different current values using different ranges of data. My head wanted to explode because I was telling them and had verified that we were getting in sufficient data.
The simplest example to illustrate this would be like the following example. Suppose that you had the following function: y = -a*x + b*x^2 + c*x^4 where a, b and c are unknown. You can measure both (x,y) and your objective is to find (a,b,c). There is some measurement noise when you get y and y is > 0
Suppose that you took data with x between 0 and 1, -a*x term is going to be the strongest and you'll probably be able to rig the equation to be always positive for some value of (a,b,c) and look about right for x between 0 and 1. But what if you took the same parameters and measured beyond [0,1] and up to [0,100]? How good would your fit be then? if your c*x^4 term had a lot of error, then the resulting graph is going to be out of whack. The (a,b,c) are going to have to be rejiggled so that the fit works well for both [0,1] and [0,100], the whole range. The values might change drastically!
And that is the problem when people try to fit limited data with higher order polynomic or non-linear functions. You could have many different functions that could describe a line segment very well, but everything falls apart when the line segment is extended, because the higher order terms were not fitted well.
I tried to explain that with all my might in Japanese. But they just didn't get it and they kept going with that the model was probably missing a few things. I even back calculated and showed reversed approximations giving a confidence interval down to 12% on the data, but alas no dice.
This is quite a problem when most people I work with are experimentalists, they just do stuff to see if things work or not. When it comes to simulation, programming or theoretical stuff, they know the basics but the real fruits are hidden in the details. Since we make some of these devices, it is important that they are characterized properly so we might have a good idea on what factors might impact the performance of these devices. Just saying that voltage is up or down isn't sufficient because you want to go after the biggest impact factors when improving things first.
Subscribe to:
Posts (Atom)