[InterMine Dev] strange database record corruption

Richard Smith richard at flymine.org
Tue Jul 7 17:24:38 BST 2015


Hi Joe,
Thanks for sending the test code, I'll try it out.

Before we go any further though please could you try merging some changes
I made to make the DirectDataLoader operate in batches:
https://github.com/intermine/intermine/pull/993

This makes the DirectDataLoader behave like the normal data loader and
batch up queries in the target database instead of running them
individually, and doesn't trigger a commit in the target database after
each object. The default batch size is 1000, you might want to set it
higher (I just noticed I left this as a constant instead of a property).

Batching queries may give a significant improvement without any jiggery
pokery with the idMap.

Thanks,
Richard.



More information about the dev mailing list