[InterMine Dev] strange database record corruption
richard at flymine.org
Tue Jul 7 17:24:38 BST 2015
Thanks for sending the test code, I'll try it out.
Before we go any further though please could you try merging some changes
I made to make the DirectDataLoader operate in batches:
This makes the DirectDataLoader behave like the normal data loader and
batch up queries in the target database instead of running them
individually, and doesn't trigger a commit in the target database after
each object. The default batch size is 1000, you might want to set it
higher (I just noticed I left this as a constant instead of a property).
Batching queries may give a significant improvement without any jiggery
pokery with the idMap.
More information about the dev