[InterMine Dev] reducing source loading times

Richard Smith richard at flymine.org
Wed Feb 8 18:07:33 GMT 2012


Hi JD,
It looks like it's the OBO edit reasoner that is taking all the time:

2012-02-07 11:06:32 INFO  org.obo.reasoner.impl.LinkPileReasoner     - 
   Total reasoner time = 2130574.717897 ms

Which is 35 minutes.  On the latest FlyMine build it took 30 seconds.
I guess this is a RAM thing.  For the FlyMine build we had 32GB heap
allocated to the Java process.  How much did you have?

The rest of the build looks like it ran fast, about 1 million objects
loaded in five minutes which is good.


Cheers,
Richard.



On 07/02/2012 16:30, JD Wong wrote:
> Sure
>
> On Tue, Feb 7, 2012 at 8:34 AM, Richard Smith <richard at flymine.org
> <mailto:richard at flymine.org>> wrote:
>
>     JD,
>     Could you send us the intermine.log from your integrate directory after
>     running a build.  This is the most helpful thing for us to investigate
>     performance.
>
>     Thanks,
>     Richard.
>
>
>
>
>
>     On 06/02/2012 19:01, JD Wong wrote:
>
>         I was wondering how the other mods speed up their builds.  I have
>         configured ant, java, and postgres accordingly without effect.
>           I was
>         hoping to get the community's advice on this aspect.
>
>         Cheers,
>         -JD
>
>         On Thu, Feb 2, 2012 at 10:11 AM, JD Wong <jdmswong at gmail.com
>         <mailto:jdmswong at gmail.com>
>         <mailto:jdmswong at gmail.com <mailto:jdmswong at gmail.com>>> wrote:
>
>             I haven't given ant and postgres enough to consume all the
>         memory
>             when running simultaneously.  Also there is plenty of free
>         memory
>             during loading.
>
>             -JD
>
>
>             On Wed, Feb 1, 2012 at 10:35 PM, Josh Goodman
>         <jogoodma at indiana.edu <mailto:jogoodma at indiana.edu>
>         <mailto:jogoodma at indiana.edu <mailto:jogoodma at indiana.edu>>> wrote:
>
>                 You need to be careful of the various memory settings
>         here.  If you
>                 set ant really high (>25% of total memory) and you are also
>                 setting Pg
>                 high you could be suffering from the two of them
>         fighting over
>                 system
>                 resources and causing the swap to get thrashed.  I would
>         run the
>                 unix
>         "free" command while you are running a load to see what is going on
>                 with memory.
>
>                 e.g.
>
>                 free -m -s 5
>
>                 If you have other processes running on this machine (tomcat
>                 instances)
>                 you also need to adjust ant and Pg to take that into
>         account.
>
>                 Josh
>
>                 On Wed, Feb 1, 2012 at 5:20 PM, JD Wong
>         <jdmswong at gmail.com <mailto:jdmswong at gmail.com>
>         <mailto:jdmswong at gmail.com <mailto:jdmswong at gmail.com>>> wrote:
>          > Hi all,
>          > Loading my Go source takes on average 2500 seconds.  I have
>                 tuned the
>          > postgres configuration paramaters to the desired values and
>                 gave ant high
>          > heap memory to no avail.  Is there a way to speed this up?
>          >
>          > -JD
>          >
>          > _________________________________________________
>          > dev mailing list
>          > dev at intermine.org <mailto:dev at intermine.org>
>         <mailto:dev at intermine.org <mailto:dev at intermine.org>>
>          > http://mail.intermine.org/cgi-__bin/mailman/listinfo/dev
>         <http://mail.intermine.org/cgi-bin/mailman/listinfo/dev>
>
>          >
>
>
>
>
>
>         _________________________________________________
>         dev mailing list
>         dev at intermine.org <mailto:dev at intermine.org>
>         http://mail.intermine.org/cgi-__bin/mailman/listinfo/dev
>         <http://mail.intermine.org/cgi-bin/mailman/listinfo/dev>
>
>
>
>     _________________________________________________
>     dev mailing list
>     dev at intermine.org <mailto:dev at intermine.org>
>     http://mail.intermine.org/cgi-__bin/mailman/listinfo/dev
>     <http://mail.intermine.org/cgi-bin/mailman/listinfo/dev>
>
>




More information about the dev mailing list