…so I derived my own in a much simpler way, and figured out a more general algorithm.  Other people have undoubtedly found the exact same method, but papers I’d found on the topic didn’t seem to state their methods much clearer than the Wikipedia page.  The issue is that I derived it in a completely different way than what’s on Wikipedia.  It’s not a matter of “fixing a mistake” in the page; by Wikipedia’s rules, this counts as original research (despite that others have probably done the same), so it’s not allowed on Wikipedia.  The only other suggestion people had was a blog post to make sure that a correct version is online somewhere, so here it is.

In a nutshell, the Jacobi eigenvalue algorithm works by one-by-one choosing the largest off-diagonal element, and rotating the two vectors associated with that element such that the element becomes zero.  Do this enough times and all off-diagonal elements will be quite close to zero (if the matrix is Hermitian), leaving the eigenvalues.  The eigenvectors come from applying the rotations to the identity matrix.  The only thing necessary to do this is to know the eigenvectors that diagonalize a 2×2 matrix, namely

where is the matrix we’re trying to diagonalize, and is the element we’re trying to eliminate.  If is Hermitian, are real, and .  However, I won’t assume that is Hermitian, since even if it’s not, this same approach can still be used to get what’s known as the Schur decomposition, namely , where is upper triangular and the eigenvalues of are on the diagonal of .  If is Hermitian, the Schur decomposition is the same as the eigendecomposition, so .  In the non-Hermitian case, it doesn’t give the eigenvectors, but those can be more easily computed afterward.  I’ll split this into two cases.

Case 1:

This case is where you don’t have two diagonal elements that are almost identical compared to the off-diagonal element.  If you did, dividing by them could cause chaos, madness, and megalomania… or just invalid results.  We start by subtracting from our 2×2 matrix, since we’re only interested in its (pseudo-)eigenvectors , and multiples of can be diagonalized using the same eigenvectors.  That basically means that adding/subtracting multiples of won’t change .  Then, we  divide the 2×2 matrix by , which also doesn’t change the (pseudo-)eigenvectors.  This gives us

For simplicity, we’ll just relabel this as

Now, if we want to make by multiplying by a unitary matrix on the right and its conjugate transpose on the right, a bit of trial and error can give you that

is a solution to that problem.  Note that the square roots are square roots of complex numbers, not necessarily real numbers.  You can verify that this works in the original case by multiplying out and confirming that the lower off-diagonal element is zero.  If is Hermitian, both off-diagonal elements should be zero.  Case 1 complete.

Case 2:

This case is where you have two diagonal elements that are almost identical compared to the off-diagonal element.  Simple solution: pretend that they are exactly zero.  You now have

The zeros make things much simpler, as we can now find

This time, since we’ve assumed that the diagonal elements are exactly the same, it probably won’t make the off-diagonal element exactly zero, but it’ll probably be negligibly small, and if it’s ever the largest at some later point, it’ll just get picked again to be eliminated, only probably without the diagonal elements being so close the 2nd time around.

Again, you don’t have to take my word for it.  You can easily verify this solution by just multiplying it out.

How did you get these?

If you’re wondering how I figured these out (after a weekend of trial and error of different methods), I knew that if the off-diagonal element was non-zero (which it must be, else there’s nothing to do), there is some non-zero component of the second element in each vector of .  I temporarily ignored the restriction of having unit vectors and set both of those components to 1, but didn’t ignore the restriction of having the two vectors being complex orthogonal (i.e. ), giving me

I know I want that off-diagonal element zero, so

Since the scales of the 2 vectors are arbitrary, including the relative scales, I scaled one by , and the other by .  Then I just normalize them both (they happened to be the same length, ), and that’s the solution above.  I did the same for case 2, only it was even simpler.  Easy… it just took a weekend to figure out what the easy way was.

Final Algorithm to Compute Schur Decomposition

V = identity matrix
while (there is an ij such that |M[i][j]| > 10^-8*|M[i][i]-M[j][j]|) {
    select ij with maximum |M[i][j]|/|M[i][i]-M[j][j]|, possibly infinite
    // Case 1
    if (|M[i][j]| < 10^14*|M[i][i]-M[j][j]|) {
        a = 2*M[i][j]/(M[i][i]-M[j][j])
        b = 2*M[j][i]/(M[i][i]-M[j][j])
        c = sqrt(1+a*b)
        L = sqrt(|a|^2 + |1+c|^2)
        Vsub = {{1+c, -conj(a)}, {a, 1+conj(c)}}/L
    }
    // Case 2
    else {
        a = M[i][j]
        b = M[j][i]
        L = sqrt(|a|+|b|)
        Vsub = {{sqrt(b), -conj(sqrt(a))}, {sqrt(a), conj(sqrt(b))}}/L
    }
    // Rotate columns i and j of M
    for (each row of M) {
        newi = M[row][i]*Vsub[0][0]+M[row][j]*Vsub[1][0]
        newj = M[row][i]*Vsub[0][1]+M[row][j]*Vsub[1][1]
        M[row][i] = newi
        M[row][j] = newj
    }
    // Rotate rows i and j of M
    for (each column of M) {
        newi = M[i][col]*conj(Vsub[0][0])+M[j][col]*conj(Vsub[1][0])
        newj = M[i][col]*conj(Vsub[0][1])+M[j][col]*conj(Vsub[1][1])
        M[i][col] = newi
        M[j][col] = newj
    }
    // Rotate cols i and j of V
    for (each row of V) {
        newi = V[row][i]*Vsub[0][0]+V[row][j]*Vsub[1][0]
        newj = V[row][i]*Vsub[0][1]+V[row][j]*Vsub[1][1]
        V[row][i] = newi
        V[row][j] = newj
    }
}
// Diagonal of M now has the eigenvalues of the input M
// If M was Hermitian, V now has the eigenvectors of the input M as columns,
//                     in the order that the eigenvalues appear in M

For those of you who don’t know, recently, North Korea bombed South Korea, killing 2 civilians and 2 marines. I think we can agree that innocent people dying is generally bad, so I won’t belabour that point.  What I’m baffled by is the BBC and Reuters, among countless others somehow equating killing 4 people with performing routine military drills.

“The Russian move followed a warning from North Korea of retaliation if the planned drills were held.”

– BBC News (underline added)

“Air-raid bunkers on the South Korean island of Yeonpyeong shook during South Korea’s exercise, which the North had vowed to repay in kind.”

– Reuters (underline added)

I’ve heard CTV and other stations talk this same way about the situation as well.  Let’s get one thing straight: North Korea “repaying in kind” for South Korea running military drills would be North Korea running its own military drills.  North Korea killing more people supposedly in response to military drills is not “retaliation”, it’s simply a unilateral attack on South Korea, no matter how North Korea tries to spin it.  None of these are quotes of North Korea; if they had been, that’d make sense, but instead, multiple news article writers around the world independently sat down and chose to write as if killing was equivalent to practicing.  Here are some real definitions of “retaliate”.

“To do something harmful or negative to get revenge for some harm; to fight back or respond in kind to an injury or affront.”

– English Wiktionary entry for “retaliate”

“make a counterattack and return like for like, especially evil for evil”

– Princeton WordNet entry for “retaliate”

News networks should not just assume that translations of North Korea’s propaganda are correct and truthful.  They’re not exactly the most reliable source!

I reject your reality and substitute my own. --Napoleon... or that guy from Mythbusters

In the past year, I’ve had more papers rejected than most PhD students are even a part of throughout their degree, but I’m not the only one.  Here’s a selection from some hilarious rejections of some of the biggest results in computer science (source: “We Are Sorry to Inform You…”).

…The author is a proponent of the so-called “structured programming” style, in which, if I get it right, gotos are replaced by indentation. Structured programming is a nice academic exercise, which works well for small examples, but I doubt that any real-world program will ever be written in such a style. … I am confident that, 30 years from now, the goto will still be alive and well and used as widely as it is today.

–Reviewer rejecting Dijkstra’s idea of structured programming, presented in ”Goto Statement Considered Harmful”

… The paper contains no real-world example to convince us that any model of practical interest can be cast in it. Quite the contrary, at first sight I doubt that anything complex enough to be of practical interest can be modeled using relations. …

–Reviewer rejecting Codd’s idea of relational databases, presented in “A Relational Model of Data for Large Shared Data Banks”

This is a bizarre paper. It begins by defining a computing device absolutely unlike anything I have seen, then proceeds to show—I haven’t quite followed the needlessly complicated formalism—that there are numbers that it can’t compute. As I see it, there are two alternatives that apply to any machine that will ever be built: Either these numbers are too big to be represented in the machine, in which case the conclusion is obvious, or they are not; in that case, a machine that can’t compute them is simply broken! …

–Reviewer rejecting Turing’s proof that there are uncomputable functions in “On Computable Numbers, with an Application to the Entscheidungsproblem”

… I don’t understand the relevance of discrete sources: No matter what one does, in the end, the signal will have to be modulated using good old-fashioned vacuum tubes, so the signal on the “channel” will always be analogical. … The author mentions computing machines, such as the recent ENIAC. Well, I guess one could connect such machines, but a recent IBM memo stated that a dozen or so such machines will be sufficient for all the computing that we’ll ever need in the foreseeable future, so there won’t be a whole lot of connecting going on with only a dozen ENIACs! …

–Reviewer rejecting Shannon’s idea of communication theory in “A Mathematical Theory of Communication”

… The scheme’s security relies on the factorization of in prime factors taking so long as to be impractical. The authors also stress that the encryption algorithm must be fast and—if their application, electronic mail, is to make sense—the algorithm should run on all sorts of machines. Let us be generous and assume that every computer user has access to a latest-generation minicomputer such as the VAX. This 32-bit machine’s speed considerations limit the choice of to . … Granted, we are seeing the appearance of so-called microcomputers, such as the recently announced Apple II, but their limitations are so great that neither they nor their descendants will have the power necessary to communicate through a network. …

–Reviewer rejecting RSA encryption, now the most prevalent encryption scheme on the planet, in “A Method for Obtaining Digital Signatures and Public-Key Cryptosystems”

The last reviewer doesn’t even have an excuse, since the Xerox Alto, with most of the technology of modern computers, had already been able to communicate through a network with 3 Mbps Ethernet for 5 years at  that point!

Anyway, I hope you enjoyed this look at one part of the failure of peer review, and when someone rejects your paper because they’re clearly either incompetent or on some absurd vendetta to discredit you, don’t fret.  It’ll probably blow up in their face in the end.   Beyond that, I think the whole idea of papers is outdated and near obsolete, since technology has advanced way beyond text-only communication of ideas… said the guy writing text to communicate ideas…

Until recently, it seems like the only such algorithms were brute force algorithms, i.e. try something and hope it works, which understandably has issues.  However, some coworkers and I have come up with ways to make non-brute-force algorithms.  I’m looking forward to see what other people can come up with, expanding upon these ideas once we’ve got them out there.

I’m especially hoping to explain things to other computer scientists down the road, since the stuff should be simple enough for interested 4th-year CS students to understand, but given how much I keep procrastinating on making video tutorials, I probably won’t get around to it as soon as I’d like.

Note that I’m not putting down Matlab, since it’s not designed to run fast; I’m just using this as an example of why it’s important to not get complacent with performance.

1. Precomputation

This is may be the largest performance difference between the two implementations, though it might be a tight race between this and the next two differences.  Precomputation, however, is very widely applicable, easy to do, and could have been applied to the Matlab code to some extent.

The basic idea is: instead of computing the same data repeatedly, compute them once and save them.

In those words, it sounds so obvious, but it encompasses a broad range of things, some of which are less obvious.  For example, in this case, I’ve precomputed the data before running the program and read it in from input files.  I’ll be running the program using the same data possibly thousands of times, and those data took longer to compute than to read in, so there’s no sense in recomputing them each time.  In particular, I’ve computed about 1,000 points on each of a bunch of curves, and the program linearly interpolates between them, which is plenty sufficient accuracy in this case.

There are of course cases where precomputation  doesn’t make sense, such as if the computation is as fast as looking up the data or if the amount of data to save is unreasonably large, but in a very large range of applications, precomputation can speed things up a ton.

2. Native Code

People will yell until they’re blue in the face that interpreted code or Just-In-Time-compiled code is usually as fast as native code (i.e. compiled to machine code ahead of time).  They’re dead wrong, and usually somewhere up the chain, they got their information from a marketing department or a study funded by Sun, IBM, etc, even though they’re probably not aware of it.  Nonetheless, I was honestly shocked that such extremely simple tests confirm this, e.g. running the following code in Java versus in C++.

int a = 12468;
for (int i = 0; i < 2000000000; ++i) {
    a /= 2;
    a *= 2;
}
// Output a

C++ compilers trivially replace the multiplication with addition, and the division with a bit shift (and a special trick because the compiler doesn’t clue in that the value is always positive or even).  Of course, a really smart compiler would realize that the entire loop is useless, since the starting number is positive and even, but none of them appear to be that smart.  Java however, will not even do the simple replacement no matter how many times the loop is run.  This is visible by comparing against the performance of:

int a = 12468;
for (int i = 0; i < 2000000000; ++i) {
    a >>= 1;
    a += a;
}
// Output a

In Java, this new code is several times faster than the previous, whereas in C++, the performance is (almost) identical between the two, since it’s already made that replacement.

In my case today, I’m not comparing against Java; I’m comparing against Matlab, which not only doesn’t compile its code ahead of time, but literally interprets the text representing the code as it’s running, similar to VisualBasic or Fortran.  That adds factors of thousands for just figuring out what the program is doing.  It just doesn’t stand a chance outside of its prebuilt libraries.  Inside its prebuilt libraries, e.g. matrix multiplication or diagonalization code written in C++, it’s fast, so one must try to use them where reasonable.

3. Customization

As much as Matlab provides some great functionality that can be a pain in the butt to implement from scratch, such as diagonalization of big matrices, it (understandably) doesn’t let you specify everything you know that may help speed up the computation.

For example, matrix-matrix multiplication where you input two matrices and get back their product is very slow because of memory caching issues.  However, if the matrix on the right of the multiplication is already transposed ahead of time, it’s much faster.  (I’ll save the explanation of why for another time.)  Matlab doesn’t know whether there’s a way to rework your code such that some matrices are pre-transposed and others aren’t, it just receives two matrices and has to spit out the product.  If you can use your knowledge of your code to do this, you can get a big performance boost.

Another example in this case is if you have good estimates of the eigenvalues and eigenvectors of the matrix you’ve got, it’s much faster to find them exactly, but Matlab’s diagonalization just uses random guesses.

Conclusion

As I mentioned at the top, the C++ implementation is mostly unoptimized, so I’m still expecting more performance improvements as I scale it up and optimize it, (since speeding up a 10ms computation isn’t all that useful.)  I’ll try to remember to report again when it’s multi-threaded and vectorized.  Nonetheless, simple changes like those above can make a huge difference in the performance of software.

The $50 Problem

I’ve got a function , which is analytic at 0 (i.e. has a valid Taylor series about ).  I want to know about the Taylor series about of the function defined as:

Specifically, is it true that:

i.e. is the derivative of evaluated at equal to the derivative of to the power of , over , all evaluated at ?  That’s all.  The first valid proof I receive showing either that the above is true for all values of or false for at least one value of will be awarded $50, and, if you’re quick enough, you’ll get acknowledged in a theory paper on quantum annealing.

Partial Results

This can be shown to be true for the first few values of :

I’ve manually confirmed that it is true like this up to and including , but each subsequent value of takes twice as long to confirm as the previous.  If there was some nice property with which to perform induction, it might collapse down to something very simple, making  it easy to prove true; I just couldn’t think of any.

Submitting Your Entry

To submit, either post a link to your proof here, or email it to me (you can find my email address at http://www.neildickson.com/) so that I can post it up if it’s correct.  Best of luck!

I’m hoping to hold more contests like this in the future,  though probably not on such mathy matters.

Solved!

As pointed out by user “achille” on sci.math, this formula is effectively equivalent to the Lagrange-Bürmann formula.  Thanks to all who tried tackling the challenge!  I’ll have to make the future contests harder, haha.

This afternoon, a pedestrian on Kingway (a busy road in Vancouver) had just started validly crossing a side street.  The white walk sign was on, and the cars on the side street faced a red light, so they must stop regardless of whether he was there or not.  Sure enough, the driver of a black SUV on the side street illegally turned right on red without stopping (1 broken law) while also texting (2 broken laws), grazed the pedestrian, and drove off (3 broken laws).  The pedestrian was more mad than hurt (he was VERY mad), but was inches from a fatal hit.  An important note: you can turn right on red, but you MUST LEGALLY STOP first and make sure that it is safe before turning right on red.

A few days ago (1 or 2), a lady started crossing Willingdon, again completely legally.  This time, it was a big delivery truck that just got a green light and decided to floor it on a right turn.  The irony is that he’d looked down at us there at the corner about to cross the road just seconds before he nearly hit the lady.  The additional irony is that I’ve seen this happen so many times that even though I was also starting to cross the road, I actually stayed back for a second to see whether he’d run us over, so I wasn’t as close to being hit.  I’ve had to jump out of the way of vehicles before because they illegally turn without ensuring the safety of the turn and would have run me over.

Back in Ottawa a few years back, a friend of mine did get hit by someone turning without looking.  He was damn lucky his bike was between him and the car.  The bike was totalled.  The pedals were mangled, the rims were bent up, and the pedal ball bearings were crushed into a powder.  He wasn’t badly hurt, but the car’s driver got out and yelled furiously at him instead of checking whether he was okay.  The car might have had scratched paint at worst, and yet this guy who had come absurdly close to breaking a cyclist’s legs had the nerve to add insult to injury by tearing a strip off him.

Don’t think I’m just picking on drivers.  Many pedestrians do illegal, dangerous things on the road too, but if they get hit, it’s probably their own fault.

Anyway, long story short: when you’re driving, please be aware of the other people on the road and look where you’re going BEFORE you go there!