[InterMine Dev] Local phytomine build

Joe Carlson jwcarlson at lbl.gov
Wed Apr 15 18:45:58 BST 2015


Hi Richard,

This is interesting. The slower rate of degradation looks promising. This is probably a combination of your better hardware and the fact that I had some postgres parameters improperly set for our last load. I hope to redo it in the next week or so to see if getting those parameters right by itself help out.

I’m also thinking about some strategies for a more brute force data loading. If I get something running, I’ll let you know. We’re getting to the position that we’re really going to have to get something much, much faster.

Joe


> On Apr 15, 2015, at 9:18 AM, Richard Smith <richard at flymine.org> wrote:
> 
> Hi Joe,
> I started building phytomine from the chado dump you provided on one of
> our servers. It's only loaded 18 genomes so far (some failed, I skipped
> them) but it does show that it started off faster than your load but also
> seems to be slowing, but perhaps less rapidly.
> 
> You can see the progress in the attached graph - in an effort to one-up
> your graph I made the marker sizes proportional to the number of objects
> loaded.
> 
> The server it's running on isn't our newest (around 4 year old 16 core
> 2.67GHz Xeon, 100GB RAM) but it does have a fast direct attached disk
> array. The build java process was set to use 10GB RAM in ANT_OPTS.
> 
> I still think the slowdown is caused by ANALYSEs running on the target
> database, once this has got a bit further I'll restart with the change I
> made to make them less frequent.
> 
> Cheers,
> Richard.
> 
> 
> <phytomine-met1.png>




More information about the dev mailing list