Just some further details, was late when I posted last night and forgot to add.
Things should work better going forward, but there could still be some duplications, unlikely, but posible. There was a bug in the recording of which items were processed, if that tweet had replies to it and certain metrics were met regarding usage of IDs / URLs and by the time the the program checked if the cache had gone. Which is why it didn't happen in my tests, as I was testing well within cache times and basic tweets.
My apologies for this, but there is no way to correct what was recorded previously. But going forward it should work better and probably you won't see any more duplications in big numbers, but, just to inform you, there is a possibility. If you do notice, please let me know and I will check and confirm it is the above case and not another bug I have missed
Regards,
Martin