[InterMine Dev] reducing source loading times

JD Wong jdmswong at gmail.com
Wed Feb 8 21:04:34 GMT 2012


ANT_OPTS has 20GB allocated to it

On Wed, Feb 8, 2012 at 1:07 PM, Richard Smith <richard at flymine.org> wrote:

> Hi JD,
> It looks like it's the OBO edit reasoner that is taking all the time:
>
> 2012-02-07 11:06:32 INFO  org.obo.reasoner.impl.**LinkPileReasoner     -
>   Total reasoner time = 2130574.717897 ms
>
> Which is 35 minutes.  On the latest FlyMine build it took 30 seconds.
> I guess this is a RAM thing.  For the FlyMine build we had 32GB heap
> allocated to the Java process.  How much did you have?
>
> The rest of the build looks like it ran fast, about 1 million objects
> loaded in five minutes which is good.
>
>
> Cheers,
> Richard.
>
>
>
>
> On 07/02/2012 16:30, JD Wong wrote:
>
>> Sure
>>
>> On Tue, Feb 7, 2012 at 8:34 AM, Richard Smith <richard at flymine.org
>> <mailto:richard at flymine.org>> wrote:
>>
>>    JD,
>>    Could you send us the intermine.log from your integrate directory after
>>    running a build.  This is the most helpful thing for us to investigate
>>    performance.
>>
>>    Thanks,
>>    Richard.
>>
>>
>>
>>
>>
>>    On 06/02/2012 19:01, JD Wong wrote:
>>
>>        I was wondering how the other mods speed up their builds.  I have
>>        configured ant, java, and postgres accordingly without effect.
>>          I was
>>        hoping to get the community's advice on this aspect.
>>
>>        Cheers,
>>        -JD
>>
>>        On Thu, Feb 2, 2012 at 10:11 AM, JD Wong <jdmswong at gmail.com
>>        <mailto:jdmswong at gmail.com>
>>        <mailto:jdmswong at gmail.com <mailto:jdmswong at gmail.com>>> wrote:
>>
>>            I haven't given ant and postgres enough to consume all the
>>        memory
>>            when running simultaneously.  Also there is plenty of free
>>        memory
>>            during loading.
>>
>>            -JD
>>
>>
>>            On Wed, Feb 1, 2012 at 10:35 PM, Josh Goodman
>>        <jogoodma at indiana.edu <mailto:jogoodma at indiana.edu>
>>        <mailto:jogoodma at indiana.edu <mailto:jogoodma at indiana.edu>>**>
>> wrote:
>>
>>                You need to be careful of the various memory settings
>>        here.  If you
>>                set ant really high (>25% of total memory) and you are also
>>                setting Pg
>>                high you could be suffering from the two of them
>>        fighting over
>>                system
>>                resources and causing the swap to get thrashed.  I would
>>        run the
>>                unix
>>        "free" command while you are running a load to see what is going on
>>                with memory.
>>
>>                e.g.
>>
>>                free -m -s 5
>>
>>                If you have other processes running on this machine (tomcat
>>                instances)
>>                you also need to adjust ant and Pg to take that into
>>        account.
>>
>>                Josh
>>
>>                On Wed, Feb 1, 2012 at 5:20 PM, JD Wong
>>        <jdmswong at gmail.com <mailto:jdmswong at gmail.com>
>>        <mailto:jdmswong at gmail.com <mailto:jdmswong at gmail.com>>> wrote:
>>         > Hi all,
>>         > Loading my Go source takes on average 2500 seconds.  I have
>>                tuned the
>>         > postgres configuration paramaters to the desired values and
>>                gave ant high
>>         > heap memory to no avail.  Is there a way to speed this up?
>>         >
>>         > -JD
>>         >
>>         > ______________________________**___________________
>>
>>         > dev mailing list
>>         > dev at intermine.org <mailto:dev at intermine.org>
>>        <mailto:dev at intermine.org <mailto:dev at intermine.org>>
>>         > http://mail.intermine.org/cgi-**__bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-__bin/mailman/listinfo/dev>
>>        <http://mail.intermine.org/**cgi-bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-bin/mailman/listinfo/dev>
>> >
>>
>>         >
>>
>>
>>
>>
>>
>>        ______________________________**___________________
>>
>>        dev mailing list
>>        dev at intermine.org <mailto:dev at intermine.org>
>>        http://mail.intermine.org/cgi-**__bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-__bin/mailman/listinfo/dev>
>>        <http://mail.intermine.org/**cgi-bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-bin/mailman/listinfo/dev>
>> >
>>
>>
>>
>>    ______________________________**___________________
>>
>>    dev mailing list
>>    dev at intermine.org <mailto:dev at intermine.org>
>>    http://mail.intermine.org/cgi-**__bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-__bin/mailman/listinfo/dev>
>>    <http://mail.intermine.org/**cgi-bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-bin/mailman/listinfo/dev>
>> >
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.intermine.org/pipermail/dev/attachments/20120208/3aef75ff/attachment.html>


More information about the dev mailing list