[InterMine Dev] reducing source loading times

Todd Harris todd at wormbase.org
Mon Feb 6 19:25:38 GMT 2012


JD, what are your build times like now?

I tend to work with test data sets during development and have just started adding full sets.  I'm curious how the build will scale.

Todd


On Feb 6, 2012, at 12:01 PM, JD Wong wrote:

> I was wondering how the other mods speed up their builds.  I have configured ant, java, and postgres accordingly without effect.  I was hoping to get the community's advice on this aspect.
> 
> Cheers,
> -JD 
> 
> On Thu, Feb 2, 2012 at 10:11 AM, JD Wong <jdmswong at gmail.com> wrote:
> I haven't given ant and postgres enough to consume all the memory when running simultaneously.  Also there is plenty of free memory during loading.
> 
> -JD
> 
> 
> On Wed, Feb 1, 2012 at 10:35 PM, Josh Goodman <jogoodma at indiana.edu> wrote:
> You need to be careful of the various memory settings here.  If you
> set ant really high (>25% of total memory) and you are also setting Pg
> high you could be suffering from the two of them fighting over system
> resources and causing the swap to get thrashed.  I would run the unix
> "free" command while you are running a load to see what is going on
> with memory.
> 
> e.g.
> 
> free -m -s 5
> 
> If you have other processes running on this machine (tomcat instances)
> you also need to adjust ant and Pg to take that into account.
> 
> Josh
> 
> On Wed, Feb 1, 2012 at 5:20 PM, JD Wong <jdmswong at gmail.com> wrote:
> > Hi all,
> > Loading my Go source takes on average 2500 seconds.  I have tuned the
> > postgres configuration paramaters to the desired values and gave ant high
> > heap memory to no avail.  Is there a way to speed this up?
> >
> > -JD
> >
> > _______________________________________________
> > dev mailing list
> > dev at intermine.org
> > http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
> >
> 
> 
> _______________________________________________
> dev mailing list
> dev at intermine.org
> http://mail.intermine.org/cgi-bin/mailman/listinfo/dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.intermine.org/pipermail/dev/attachments/20120206/0b3e86e5/attachment.html>


More information about the dev mailing list