Hessian Matrix

Learning!
I just love peeling back the onion. Now that I have the classic gradient descent working, what is the next step? 
BFGS apparently. http://en.wikipedia.org/wiki/BFGS_method

And I learn about a new matrix: The Hessian Matrix, (which I have been seeing mentioned a lot lately, but didn't know what it was).
http://en.wikipedia.org/wiki/Hessian_matrix

So, my intuition (before researching this) is: We are using Gradient Descent to move to the solution (where the first derivative = 0). What would be really handy is if we had the second derivative to know what our slope is, and if it's very steep, then we can make a bigger jump… Or actually if we make a small change in X, it's going to make a large change in Y due to the steep slope..

And presto, the Hessian Matrix is actually the Second Derivative. 
Also of note BFGS, needs NxN elements for the Hessian, where N is the number of parameters in the model, and there is also L-BFGS for limited memory. 

So sizing this up: For 100,000 variables (which is an epic-non-human-readable model), we would need 80GB, but I'm using less than 1K variables, so 8MB should do.

-JD

Online Incremental and Decremental Stats

Thinking about implementing this in forwards and reverse.

http://www.mymathforum.com/viewtopic.php?f=44&t=14057

http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

Now to figure out skew and kurtosis... Also I note that the wikipedia article has parallel version so I can do it in parts. (Which would come in useful on a hadoop cluster for sure).. Also I should have paid more attention to recurrence relations in math class..




public static class Variance {
        private long numEntries;
        private double mean;
        private double secondMoment;

         public void add(double value ) {
            this.numEntries++;
            double delta = value - this.mean;
            this.mean = this.mean + (delta/this.numEntries);
            this.secondMoment = this.secondMoment + (delta * (value - this.mean) );

          }


public void remove(double value) {
            double oldMean = mean;
            this.mean *= numEntries;
            this.numEntries--;
            this.mean = (this.mean - value) / this.numEntries;
            this.secondMoment -= (value - oldMean)*(value - this.mean);

}


public double getVariance() {
            return( this.secondMoment / (this.numEntries - 1 ) );
        }
}

Regression

Well, It's been at least 15 years since I first gained an interest in Machine Learning, Neural Nets, etc. I have many books on the subject that really I couldn't quite figure out what all was going on. Too many Summations, subscripts and superscripts to take in. I wonder if I have some sort of dyslexia in this regard? But Huzzah! At last I have finally taken the time to write some very simple software that has opened up all of those formerly arcane subjects:

A very simple and basic Linear Regression which I hope will be the beginning of many understandings to come.

https://github.com/fattygustave/mltest

public DoubleMatrix1D descent(double         alpha,
                              DoubleMatrix1D thetas, 
                              DoubleMatrix2D independent,
                              DoubleMatrix1D dependent ) {
         Algebra algebra     = new Algebra();
         
        // ALPHA*(1/M) in one.
        double  modifier    = alpha / (double)independent.rows();
        
        //I think this can just skip the transpose of theta.
        //This is the result of every Xi run through the theta (hypothesis fn)
        //So each Xj feature is multiplied by its Theata, to get the results of the hypotesis
        DoubleMatrix1D hypothesies = algebra.mult( independent, thetas );
        
        //hypothesis - Y
        //Now we have for each Xi, the difference between predicted by the hypothesis and the actual Yi
        hypothesies.assign(dependent, Functions.minus);
        
        //Transpose Examples(MxN) to NxM so we can matrix multiply by hypothesis Nx1
        //Note that the Transpose is constant time and doesn't create a new matrix.
        DoubleMatrix2D transposed = algebra.transpose(independent);

        DoubleMatrix1D deltas     = algebra.mult(transposed, hypothesies );

        // Scale the deltas by 1/m and learning rate alhpa.  (alpha/m)
        //deltas.assign(Functions.mult(modifier));
       
        //Theta = Theta - Deltas
        //thetas.assign( deltas, Functions.minus );

        // thetas = thetas - (deltas*modifier)  in one step
        thetas.assign(deltas, Functions.minusMult(modifier));

        return( thetas );

    }

Selby

I was sitting at home and had a profound experience. I experienced, in all of my Being, that someday I was going to die, and it wouldn't be like it had been happening, almost dying but somehow staying alive, but I would just die! And two things would happen right before I died: I would regret my entire life; I would want to live it over again. This terrified me. The thought that I would live my entire life, look at it and realize I blew it forced me to do something with my life.

The Edge

Somedays I reflect on people that survive impossible situations. They fall out of airplanes with only a broken arm, they walk away from car crashes, fires, and explosions. I marvel at the resiliency of life, and the abundance of second chances. Other days .. less optimistic I guess .. I'm saddened by those that are lost to things mundane, or dangers not so apparent. And for a moment I think my life has been safe and uneventful, progressing mostly without event, and then I consider the uncountable number of ways my life could have gone horribly wrong (and could still). So many checks I had written with bad choices, and so much the balance that has expired uncollected. You think the Devil actually shows up with a contract? You think you will be able to laugh in his face and walk away the victor? Not so easy as all that.

http://www.cnn.com/2013/02/04/health/synthetic-marijuana-irpt/

-JD