I've just tried it on my humble 8 GiB RAM (32-bit, so actually 4 GiB per process), and It fails only at 43.7 million data points due to a bad alloc (out of memory). And this is with Qt 5.3, so using the red-black-tree map structure. Of course having all of them on screen is sluggish as hell, but it doesn't throw a bad alloc if below 43 Million. At that point, the process uses 2.8 GiB of memory. So I'm not sure what's going on in your case, it sure isn't expected.
A change that will come in Version 2.0 of QCustomPlot is the trimming of the QCPData container to remove all the error variables that are not used in the majority of cases and thus waste memory (errorbar functionality will be preserved via an additional plottable). So you could try this to reduce the memory footprint of the graph. Strip off the
double keyErrorPlus, keyErrorMinus;
double valueErrorPlus, valueErrorMinus;
part, reducing your data point memory footprint by two thirds. You'll also have to get rid of/comment out the few code sections that use those variables. I suggest just removing the above lines from QCPData and then moving from syntax error to syntax error until it compiles. No hidden traps there.
Other hard-core users of QCP have converted the entire library to be float-based, gaining some speed and halving the memory footprint at the cost of number precision and maximum/minimum magnitude. But that might be a bit overkill to sort out your situation.