But How Can I be Sure my Widget is Actually Working?

But How Can I be Sure my Widget is Actually Working?

Welcome to the world of statistics, where we try to convince ourselves and our employers/clients/friends/wives that the projects we have spent hours and hours on are actually worthwhile! We will see by the end of this article just how you can too can demonstrate with reasonable certainty that your very own predictor/estimator/widget is having a positive (or negative effect) on the process it is being applied to.

Yes, we're talking about hypothesis testing!

What hypothesis?

The hypothesis that results from the question about whether our widget has a positive (or negative) effect on the process, of course ! If we want to know whether the widget has a positive or negative effect then we must hypothesize the opposite (that there is no effect, or that the widget and the normal process are the same) and try to prove it wrong! This may sound counter-intuitive but one limiting factor about statistics is that you can't prove a statement true, you can only use it to reject a statement (though your're not disproving it).

For example, for my stock market predicting widget, I want to know how I can be 95% confident in my belief that my estimated prediction accuracy of 57% is not just random chance (luck). If you've ever studied the stock market you've probably heard the cliche about how someone let monkeys pick stocks, or threw darts at a dartboard and the resulting random selection of stocks did better than "such and such" a famous money manager. Well, I don't want to end up being the money manager that gets beat by a monkey, that's for sure!

First, in statistics, we are always estimating parameters. In this case we are only estimating the ability of my stock predictor to make correct stock predictions. Because I can't, or wont, test the predictor on every stock ever I can't know it's true 100% ability to make predictions. But this is where the beauty of mathematical statistics comes in...we don't have to! I can estimate the stock's accuracy on a relatively small number of days within the stock's history and then use the principals of confidence intervals to establish a level of confidence or belief that my predictor is better or at least not eqaul to a random predictor.

Secondly, we want to know something about how we expect the process to function without the help of our widget. For my stock widget, I want to know if my predictor is significantly different than a random predictor, i.e. a monkey throwing a dart at a dart board filled with stock predictions (if monkeys could do that).

So how do we know what our process will do without our widget? Easy! We simply sample the output! Sometimes we can do this theoretically, as in the case of my stock market predictor.

My stock predictor only makes predictions on whether the stock should be bought or sold. I can say that the accuracy of my stock predictor, which comes in the form n correct predictions out of N attempts, looks a lot like a Binomial distribution! Actually, Binomial distributions look a lot like Normal/Gaussian distributions ...otherwise known as the Bell curve. A Binomial distribution can be produced with the following 4 steps:

1.) flip a coin 100 times and write down how many times you got heads - we'll call this a "coin flip trial" with a heads population of "p-heads".

2.) we do 1000 "coin flip trials" - yes it will take a while

3.) after the "coin flip trials" are finished we make a chart and plot each unique value of "p-heads", and the number of times we got each one of the "p-heads".

4.) We stand back and marvel at our new representation of the Binomial Distribution - it should look this.

This is what would happen if I randomly chose whether the stock would go up or down. It would achieve 50% accuracy on average.

If you get an evenly balanced coin you will notice that your most popular choice for "p-head" is 50. You will also notice that most of the values fall within the range 45 and 55.

Now, let's say that the evenly balanced coin is a random stock predictor. It will only predict half of all possible stock moves correctly... i.e. you lose as much as you make over time. But when I run my stock predictor on a random sample of the market it tells me that I got 57% of the predictions correct!. Since this estimated accuracy is only on a small random portion of the market, how do I know that I didn't just get lucky? How do I know I'm not a monkey (this may be an altogether different question :-)?

Going back to my hypothesis, I need to evaluate the claim that (Accuracy_random = Accuracy_mystockwidget). But this is the same as evaluating (0 = Accruacy_mystockwidget - Accuracy_random) Now, we know that the accuracy of the random predictor will be distributed according to the picture above. Incredibly, my stock predictor should actually follow the same pattern, only shifted over to the right slightly. But here's the even more incredible thing, if we subtract the two variables, and plot the results the associated relative frequencies, it will also look like the above!

If the resulting distribution contains 0 in a centered 95% selection of it's values about its mean, then we will be forced to make the satement, "We fail to reject the hypothesis that Theo's stock widget is equal to a random monkey predictor." However, if 0 is not found about this center 95% of the distribution, then I can proudly say that I reject the claim that they are equal in favor of the claim that my predictor is different from a coin-flippining monkey (and I will later go on to say that of course it is much better!). So what do the numbers say? Let's take a look at another sweet chart!

And so we see that from a statistical standpoint, my stock predicting widget is significantly different from a coin-flipping-dart-throwing monkey predictor!

Isn't statistics cool?

note: I haven't yet acheived 57% accuracy yet, so hold your horses and your money until I do :-)

Stock Market Analysis Tools

Stock Market Analysis Tools

For the last 5 months (basically since I finished my Master's thesis) my stock market prediction software has been on hold. I was making great progress on it and thought that I had finally cracked the problem last September. However after running a real time trial that essentially paper traded all the stocks on the S&P 100 on a daily basis, my theoretical accuracy of 60-70% prediction rates turned out to be a bit of a wash. In short, after three months I neither made money, nor lost money. But I did spend about 1000 dollars a day in trading fees (err... virtual dollars that is).

The upper line is my theoretical profit on the stock CAT using predictions...the lower line is the value without predictions. The triangles are predicted short sells, while the circles are predicted long buys, see why I was excited?

I racked my brain and poured through my code. After a month of searching through the code and comparing the real time predictions to the simulated predictions my conclusion is that my framework for building and testing my learning models is sound. However the data that I was using to train the models was not. Tracing it down even further I began to realize that the functions I had written to compute the technical analysis indicators (see note 1) were incorrectly programmed and brought inconsistencies to the data. Thus the data for the training sets was inconsistent with the live data... my models were learning Portuguese while the stock market was speaking Spanish.

Not wishing to rewrite all those functions again I set out looking for some open source software to compute the technical indicators. Amazingly, I could not find a single Matlab, Python, Java or any language package that did such a simple task. I put it on the back burner and pursued other things for a while.

But all is not lost! Yesterday while searching through packages on the statistical computing language R I finally found one that did it! So now the task is for me to write some functions to take my data, use the preexisting functions in Rmetrics to transform it, and then rerun it back through my prediction engine....and start another real time trial! The website that this cool package can be found is... Rmetrics .

Enjoy, I know you all are just dying to try out both R and Rmetrics!

Note 1 - Technical Analysis Indicators are nothing more than non-linear transformations of the price data which are meant to bring out certain aspects of the price. For instance, a 200 day moving average indicator attempts to show a trend in the data using a 200 day smoothing transformation (actually just an average). For SVM's (the basis of my prediction algorithm) non-linear transforms are like glasses...they really bring clarity to an otherwise very fuzzy data set.

A Random Walk Through the Stock Market

A Random Walk Through the Stock Market

Have you ever taken a random walk in the park? If you say yes, then chances are, you are incorrect. Nearly everything we do has purpose and direction to it...if only because our minds have trouble doing anything without it.

The Stock Market exhibits a mathematical phenomena we call a Random Walk. A truly random walk in the park would mean that one starts off in the middle (or somewhere else), spins a four sided dice and decides to take a step forward or backward, left or right, based on the dice roll. Now aside from the question about whether a dice roll is truly random we will say that the path taken by our dice rolling park-walker is random. It may even look like this...

So how does this relate to the Stock Market? If we look at the stock market as a random walk in a 1 dimensional park, or a tight rope, than the price is randomly walking from low to high....or at least it is very nearly randomly walking. Google has been marching upwards for a while, while Microsoft has been backpedaling for a while now. But for the most part we can look at a stock as a very indecisive tight rope walker, he flips a coin and takes a step forward, flips the coin again and takes another step forward, flips the coin a third time and goes back, etc. etc.

So how can we take advantage of a random walk? Let's say that you and a buddy go to see the random tight rope walker. Let's say that your friend likes to gamble. He says to you....I'll bet you $1 that every time the tight rope walker flips that coin, you can't predict whether he will go backwards and forwards. Unfortunately, your friend is unaware that you are an expert in coin flip pattern recognition, and you are able to predict heads or tails with 55% accuracy. As the hours wear on you begin to slowly increase your profit. In fact, if you plotted your profit, it may look something like this...

Thus, a random walk not only describes the movement of the price of a stock, but for a day trader, it describes the value of his bank account. Therefore, one can nearly always make money in the stock market, provided one can predict the next general movement of the market with only a slightly better than 50% accuracy level.

So the real question is....why spend time trying to predict whether my favorite stock will shoot up 20 points in the next month when all I need to focus on is predicting the up and down movements and letting the laws of probability and Random Walk work in my favor?

Reasons Why

I have started this blog for a few reasons. I hope that my posts are useful and productive for myself and anyone else who stumbles across them. Hopefully I will have at least one post a week on one of the following topics:

1.) Green Technology - all things that give us free energy, or help to conserve energy without making my life a drag.

2.) Math, Statistics, Machine Learning and the Stock Market - I'm currently joining the ranks of modern day alchemists and trying to predict the stock market on a day to day basis. I agree that if successful I will eventually defeat myself because I will sway the market one way or the other. But a boy can still dream

3.) Philosophy - any philosophical musings and ideas. The nature of the mind. Psychology and all of the "soft" sciences.

4.) Entrepreneurship - I'm excited about getting into the business world. I'm trying to focus on non-pie-in-the-sky million dollar eyes and instead focus on how to take a non-sexy idea and make a profit from it.