I am posting this in case anybody debugging something needs to find it -- I did find mention of it on some Java related site, but nothing conclusive. This may affect VC2010, too, but I haven't tested it.

While VC 2005/2008 targeting x64 generates SSE code for floating point code, fmod() still uses the x87 FPU, and more importantly it assumes that the divide by 0 exception flag is clear going in (meaning if it is set prior to the call, the call will throw an exception or return #.IND regardless of the input). Apparently they assume that since the compiler won't possibly generate code that would cause the divide by 0 floating point exception flag to be set, then it would safe to assume that flag will always be clear. Or it could be a typo. If you use assembly code, or load a module compiled with another compiler that generates x87 code, this can be a huge problem.

Take this example (hi.cpp):

#include <stdio.h>
#include <math.h>

extern "C" void asmfunc();

int main() {
  asmfunc();
  printf("%f\n",fmod(1.0,2.0));
  return 0;
}

and hihi.asm (compile with nasm -f win64):

SECTION .text

global asmfunc
asmfunc:
  fld1
  fldz
  fdivp
  fstp st0
  ret

Compiling this (cl.exe hi.cpp hihi.obj) and running it does not print 1.0, as it should.

The solution we use is to call 'fclex' after any code that might use the FPU. Or not use fmod(). Or call fclex before fmod() every time. I should note that if you use ICC with VC200x, it doesn't have this problem (it presumably has a faster, correct fmod() implementation).