M95 Cursor randomly "slows down"

I'm having a problem that I have been trying to figure out - it seems after moving the mouse around a while the cursor seems to slow down or jumps like it isn't tracking properly.

If I do the surface quality test (on a brand new Corsair mousepad even) it varies wildly from the lowest to about mid range, even going in circles over the same area on the pad.

If I put it on a highly reflective surface it seems to work ok, but even then the quality varies from low to high.

I've tried different mouse pads, changing the dpi, hardware vs software mode.

If I wait eventually it gets better and but will slow down again, or if I unplug it and plug it back in it is ok for a short while.

I've already had to RMA the first one of these I had for a different issue, I really hope its not a hardware issue.