Stock Market Analysis Tools

Stock Market Analysis Tools

For the last 5 months (basically since I finished my Master's thesis) my stock market prediction software has been on hold. I was making great progress on it and thought that I had finally cracked the problem last September. However after running a real time trial that essentially paper traded all the stocks on the S&P 100 on a daily basis, my theoretical accuracy of 60-70% prediction rates turned out to be a bit of a wash. In short, after three months I neither made money, nor lost money. But I did spend about 1000 dollars a day in trading fees (err... virtual dollars that is).

The upper line is my theoretical profit on the stock CAT using predictions...the lower line is the value without predictions. The triangles are predicted short sells, while the circles are predicted long buys, see why I was excited?

I racked my brain and poured through my code. After a month of searching through the code and comparing the real time predictions to the simulated predictions my conclusion is that my framework for building and testing my learning models is sound. However the data that I was using to train the models was not. Tracing it down even further I began to realize that the functions I had written to compute the technical analysis indicators (see note 1) were incorrectly programmed and brought inconsistencies to the data. Thus the data for the training sets was inconsistent with the live data... my models were learning Portuguese while the stock market was speaking Spanish.

Not wishing to rewrite all those functions again I set out looking for some open source software to compute the technical indicators. Amazingly, I could not find a single Matlab, Python, Java or any language package that did such a simple task. I put it on the back burner and pursued other things for a while.

But all is not lost! Yesterday while searching through packages on the statistical computing language R I finally found one that did it! So now the task is for me to write some functions to take my data, use the preexisting functions in Rmetrics to transform it, and then rerun it back through my prediction engine....and start another real time trial! The website that this cool package can be found is... Rmetrics .

Enjoy, I know you all are just dying to try out both R and Rmetrics!

Note 1 - Technical Analysis Indicators are nothing more than non-linear transformations of the price data which are meant to bring out certain aspects of the price. For instance, a 200 day moving average indicator attempts to show a trend in the data using a 200 day smoothing transformation (actually just an average). For SVM's (the basis of my prediction algorithm) non-linear transforms are like glasses...they really bring clarity to an otherwise very fuzzy data set.

Sahara Desert Not a Viable Source of Solar Power

Sahara Desert Not a Viable Source of Solar Power

In one of my last posts I talked about the math behind using solar power. In the end I concluded that the Sahara Desert could power the earth 211 times over. After sending out this link to a lot of family and friends the number question/objection I received back was, "But putting solar panels in the Sahara Desert is really impractical." I would first like to say that the Sahara Desert was just a comparison point to show that even with a small fraction of the available land mass (all continents have unused deserts) we can power the world.

Now it is quite true that putting a lot of solar panels in the Sahara Desert is very impractical. As my Uncle Steve said, "Who the heck are you going to get to work in the middle of the blazing hot desert?!" And as my sister said, "But how are you going to get all the power all the way back to America?"

But this brings up a very interesting and critical point: this is exactly what we are doing in Iraq, Iran, Kuwait, the UAE and Saudi Arabia this very day. There are apparently many people willing to work in the blazing hot deserts to set up oil wells, and refining stations. Furthermore we are shipping millions of barrels of the oil all the way across the ocean in giant ships. I will also point out that several ships have unfortunately crashed or been sunk in the recent past releasing their cargo onto the beaches of Spain and Alaska (to name a few) creating some of the worst ecological disasters on record.

So even though I do believe that covering the Sahara in solar panels is a bad idea, I also think it's paradoxical that most of the public would react with a similar objection, "It's just too impractical!" We are already going to greater trouble to get oil out of the ground than it would take to install panels in the Sahara and run cables across the Atlantic. Clearly, the best solution is to install high efficiency solar capturing devices on businesses and homes across the world thus eliminating the need for a large scale electrical grid and placing the "power" back in the hands of the people.

update: Since posting the original article 5 minutes ago I read the comments from my friend Ray who says that in China they have cool technology such as: electic bicycles, solar hotwater (he hasn't paid a dime for hot water since he moved there he says), and toilets that flush with the option of 1 liter (for a #2) and half a liter (for a #1). Go China!

A False Dilemma Between Water and Energy?

A False Dilemma?

This article in the Christian Science Monitor points out the growing concern in the US Southwest about water shortages. While droughts in the desert are nothing new, the concern is that water usage by electric utilities accounts for about 60 percent of all non-farm water usage. In a region that is growing by leaps and bounds this could be a problem. As more people move in, more electricity is demanded, thus more water to produce the electricity is required. But let's not forget that we drink water and bathe in it, wash our cars and water our lawns with it too.

So how do we produce enough electricity to maintain our lifestyles while keeping enough water available to drink, shower, bath and keep all the little animals who also need water happy? Well, if you read my previous post, you will know my opinion. Solar power! The sun provides us very close to free energy. All we have to do is catch it!

The Mathematics of Free Energy

The Mathematics of Free Energy

Some would note that I am a bit idealist about getting off the grid and Green energy. Some might say it's impossible and rather impractical. However, I believe that the mathematics behind it really "adds up". Here's a look at the arithmetic behind it all.

First the energy usage of the world per year: from ecoworld.com we have the following estimated energy usage in 1995:

316,000,000,000,000,000 BTU = 92,614,302,461,899 KW hrs

The math: The sun pumps out 1.3 KW hrs/m^2 of energy in outer space. The actual sunlight hitting the earth's surface varies throughout the year but let's say it averages 40% of the outer space value (which is a little on the low end). The earth has a land area of about 148,939,100 km^2. If we can harness that energy at 50% efficiency (high for existing technologies), then the following tells us how much energy we can catch in one hour on average.

1.3 (kW hrs/m^2)*1000000 (m^2/km^2)*148,939,100 (km^2) * .50 (efficiency) * .4 (amount reaching earth) = 38,724,166,000,000 kW hrs

More simple math shows that in only 2 hours the earth receives enough energy to power it for the rest of the year.

However let's say that we wished to avoid covering the whole earth in solar panels (after all I do like the green grass and nice trees...the minimum land area that we would need to produce to catch the energy required in one year would be:

Lmin = 92,614,302,461,899 KW hrs/(1.3 (KW hrs/m^2)*1000000 (m^2/km^2)* .50 (efficiency) * .4 (amount reaching earth) *24 (hrs/day)*365 (days/year)) = 40663 km^2

Thus with only .02 % of the earth's available land (or presumably oceanic) surface we could generate enough energy to totally power the earth for a year. Even if you factor in an extra 50% energy usage growth since 1995 and another 50% for future usage we would still only need to cover less than .05% of the earth in solar panels.

To put this in perspective, covering the otherwise unused Sahara desert (86,000,000 km^2) with solar panels would generate enough power each year to power 211 earths.

The proof is in the numbers, there is absolutely no reason to keep paying for oil, nuclear or anything else. Let's put up some solar panels, or windmills or invent something even better!

Why I want to Live off the Grid - Part II, Not Living in the Stone Age

Why I want to Live off the Grid - Part II, Not Living in the Stone Age

When it comes to the practicalities of living off the grid there are many options, and just as many enthusiasts willing to help. The Internet is home to a growing community of DIY green enthusiasts who's advice the off-gridder will most appreciate. A Good starting place is Treehugger.com.

Thoreau's explains in his classic book Walden that the bare minimum a person needs to live a comfortable life is enough food to survive, clean water and a basic shelter that provides protection from the environment. All else, he says, merely allows someone to die in excess comfort. To Thoreau's requirements I add that it is imperative to have an energy resource. For off-grid purposes it must be renewable and as independent as possible.

But where to start? Let's start with a basic inventory our modern lifestyle.

Electricity - Electricity the activity of electrons that is wired into our houses at 110-120 Volts, and alternates current at a rate of 60 times per second. In England and most the world, electricity is set to 220-230 Volts, and alternates current 50 times per second. Solar Panels generally produce electricity at either 12 or 24 Volts and does not alternate current. In order to transform Solar Powered electricity to household electricity an inverter is necessary. Inverters are complicated, see wikipedia.org. Interestingly electrons in electricity move at the speed of light, however the common notion that electricity "flows" through your house is actually incorrect. Electrons can drift through the wires, but for the most part actually stay put in the wires. More or less electrons actually sort of vibrate around with greater or lesser intensity.

Water - Clean water is a necessity. Imagine trying to make a good cup of coffee with muddy water...bleh! Aside from the clarity of water it is imperative that it contains no cysts, bacteria, fecal matter, protozoa or other dangerous and alive organisms. Water is still pumped into our houses the old fashioned way, pipes and valves from a main source. However, if you live in a house off the water grid you will be responsible for pumping it out of the ground, or purifying it from rain...or if the conditions are right collecting it from the air via fog collectors !

Heat - House heating is usually accomplished via fossil fuels. Generally fossil fuels are burnt as efficiently as possible and the resulting warmth is blown throughout the house. Lately there is a resurgence of wood and wood pellet stoves for heating a house. Aside from being more earth friendly than fossil fuels, they are much more rustic and Romantic!

Insulation - Our houses are insulated well..keeping the hot air in in the winter and the cool air in in the summer. This is a key ingredient to a comfortable house.

Air Conditioning/Cooling - In the summer most houses get really warm, thus some form of cooling is required. In Dry Climates evaporative coolers use the principle that hot dry air can be made cool be absorbing the heat in water. In more humid climates Freon is expanded and contracted in order to capture heat from the air; our vehicles use this method as well. There are more methods for cooling air including acoustic cooling!

Food - Most of the food we consume is produced en mass from farms far away from our houses. Food however can be grown efficiently and with much benefit in our own back yard!

Transportation - Nothing defines America like cars. We love cars and we love going places. And we love cheap gas. Unfortunately some of that may need to be rethought as gas prices keep on climbing. Interestingly, the largest producers of oil in the middle east are some of the largest investors in renewable energy...hmmm....interesting.

Appliances - I love appliances. I love to be able to wash my clothes, and dry them quickly, wash my dishes with little effort, put 2 cups of raw rice in the rice cooker and have steamed rice 20 minutes later. I love being able to have cold ice-cream in the refrigerator. I love not having to plug my computer in to use the Internet. Unfortunately all appliances especially washers and dryers use incredible amounts of water and electricity.

Communication - The Internet is perhaps the best invention in Modern History. In fact, the guy who invented packet switching, which makes the Internet possible, was recently inducted into the inventors hall of fame. All devices that enable us to talk and communicate of long distances are not only incredible, but incredibly useful.

Entertainment - Nothing says humanity like entertainment. In a way, entertainment is the lifeblood of our culture. Entertainment is really just story telling. And story telling tells us who we are and what we are doing.

These are the basics of our modern lifestyle. Certainly one could find many more essentials to our modern life, but without the above, we would hardly recognize our lives as modern.

In the next few articles I will explore each topic in more detail and offer some practicle suggestions.

Links to the first two entries in this series

Why I want to Live off the Grid - Part I, Interdependence
Why I want to Live off the Grid - Introduction

Why is there something, instead of nothing?

Why is there something, instead of nothing?

Philosophers are always pondering such questions like why is there something instead of nothing (thanks Jason Fraze for the question prompt) I don't think that this is really an answerable question.

However I would like to ask the question is there such a thing as nothing? In other words is there a place in the universe that is totally devoid of an existent thing?

After spending much too long on the subject I have concluded that no, nothing cannot exist. (In my proof I will prove that non-beings cannot exist, nothing is a subset of the non-beings set).

Proof:

Suppose that there are actually two possible classes of things...beings and non-beings. Assume that a set of non-beings exists. These non-beings are defined in that they are the compliment of the set of all things that are beings (In other words non-beings have no existence). But the set of non-beings is definable. And all things that are definable have a reality (however limited their reality is). But this is a contradiction in that non-beings must not have a reality for they are the compliment of all things real. Therefore by the law of non-contradiction, a non-being, or a nothing cannot be said to have existence or reality.

So what does this mean in practical reality? I'm not sure, but it brings a few thoughts to my mind that there cannot be such a point in history that nothing existed. So at the very least the Big-Bang theory falls short in explaining the universe. For even prior to the Big Bang, there was something. And before that there was something, so on ad infinitum. What is the answer to this "infinite regress" as Aquinas called it? God.

Why I Want to Live off the Grid - Part I, Interdependence, or how the TV is the Devil

Why I Want to Live off the Grid - Part I, Interdependence, or how the TV is the Devil

In High School I quickly became a teachers pet for a paper I wrote called "Television: The Work of the Devil?". Most people who know me, know that I am a well-meaning Christian, and I must first say that this paper was not about the moral evils of television content (though indeed I do find most television content unsavory). It was rather a look at how television has influenced our culture, our selves and our sense of well being. But first and foremost it looked at the ways in which television is a mind and creativity destroyer.

Being on the grid necessarily implies dependence on a larger framework for sustenance. Now to be clear we must reaffirm that one can never be fully independent. No man is an island to quote John Donne. In fact let's review his poem here; it speaks volumes about the very core of humanity.

"All mankind is of one author, and is one volume; when one man dies, one chapter is not torn out of the book, but translated into a better language; and every chapter must be so translated...As therefore the bell that rings to a sermon, calls not upon the preacher only, but upon the congregation to come: so this bell calls us all: but how much more me, who am brought so near the door by this sickness....No man is an island, entire of itself...any man's death diminishes me, because I am involved in mankind; and therefore never send to know for whom the bell tolls; it tolls for thee." - Meditation XVII

Knowing that we are all interdependent and connected means that in some way we have a very sincere and natural need for human interaction. How sad is it then that the average American spends about 5-7 hours watching television a day? In fact, if we sleep for 7-8 hours and work for 8-9 hours, that leaves us with about 1-3 hour of totally free interaction with our friends, our spouses, people we don't know or haven't yet met, much less our grandparents and other distant relatives. To put this in perspective, 5-7 hours a day of television watching a day works will end up being 20-30% of one's lifetime.

At this point, it strikes me that when we talk about off the grid living, we are really talking about a way to become more interconnected with life, humanity, and the environment. An off the grid lifestyle is paradoxical then in that our images of an off-the-gridder is a solitary person in a cabin, by himself, who is kind of nutty and has a very long and disgusting beard (if he is a male of course). However, on the grid people seem to live in nice suburbs with all the modern conveniences and are quite content with the status quo. Thus the paradox is that not having to depend on the modern infrastructure (independence) will lead us to greater interdependence (an independence conditional on other more immediate human beings). However the dependency on modern conveniences has led to a great deal of Independence in the on-grid person...so much so that they often lead lives of quiet desperation and loneliness.

I believe that the first step to getting off the grid of mental dullness, the grid of contentment with the status quo, the grid of normality and blandness is to get rid of our televisions...or at the very least to get rid of cable television and to remove the television from the prominent spot in our living rooms. I think that real life has enough challenges, adventure and mystery to never get bored.

I hope in the next few posts about getting off the grid that we will begin to further understand what it means that get off the grid, and why it is the best thing we can do. We will further understand that becoming independent will lead to interdependence because, in essence, getting off the grid means changing our dependencies to a much more local set of people...something known as interdependence.

A Random Walk Through the Stock Market

A Random Walk Through the Stock Market

Have you ever taken a random walk in the park? If you say yes, then chances are, you are incorrect. Nearly everything we do has purpose and direction to it...if only because our minds have trouble doing anything without it.

The Stock Market exhibits a mathematical phenomena we call a Random Walk. A truly random walk in the park would mean that one starts off in the middle (or somewhere else), spins a four sided dice and decides to take a step forward or backward, left or right, based on the dice roll. Now aside from the question about whether a dice roll is truly random we will say that the path taken by our dice rolling park-walker is random. It may even look like this...

So how does this relate to the Stock Market? If we look at the stock market as a random walk in a 1 dimensional park, or a tight rope, than the price is randomly walking from low to high....or at least it is very nearly randomly walking. Google has been marching upwards for a while, while Microsoft has been backpedaling for a while now. But for the most part we can look at a stock as a very indecisive tight rope walker, he flips a coin and takes a step forward, flips the coin again and takes another step forward, flips the coin a third time and goes back, etc. etc.

So how can we take advantage of a random walk? Let's say that you and a buddy go to see the random tight rope walker. Let's say that your friend likes to gamble. He says to you....I'll bet you $1 that every time the tight rope walker flips that coin, you can't predict whether he will go backwards and forwards. Unfortunately, your friend is unaware that you are an expert in coin flip pattern recognition, and you are able to predict heads or tails with 55% accuracy. As the hours wear on you begin to slowly increase your profit. In fact, if you plotted your profit, it may look something like this...

Thus, a random walk not only describes the movement of the price of a stock, but for a day trader, it describes the value of his bank account. Therefore, one can nearly always make money in the stock market, provided one can predict the next general movement of the market with only a slightly better than 50% accuracy level.

So the real question is....why spend time trying to predict whether my favorite stock will shoot up 20 points in the next month when all I need to focus on is predicting the up and down movements and letting the laws of probability and Random Walk work in my favor?