[InterMine Dev] reducing source loading times

JD Wong jdmswong at gmail.com
Fri Feb 10 15:54:37 GMT 2012


In other words I set ANT_OPTS="...  -Xmx 20000m in my .bashrc file.  20G
should be a good amount, and since these values transfer to the java calls
that ant makes this is a strange problem indeed...

-JD

On Wed, Feb 8, 2012 at 4:04 PM, JD Wong <jdmswong at gmail.com> wrote:

> ANT_OPTS has 20GB allocated to it
>
>
> On Wed, Feb 8, 2012 at 1:07 PM, Richard Smith <richard at flymine.org> wrote:
>
>> Hi JD,
>> It looks like it's the OBO edit reasoner that is taking all the time:
>>
>> 2012-02-07 11:06:32 INFO  org.obo.reasoner.impl.**LinkPileReasoner     -
>>   Total reasoner time = 2130574.717897 ms
>>
>> Which is 35 minutes.  On the latest FlyMine build it took 30 seconds.
>> I guess this is a RAM thing.  For the FlyMine build we had 32GB heap
>> allocated to the Java process.  How much did you have?
>>
>> The rest of the build looks like it ran fast, about 1 million objects
>> loaded in five minutes which is good.
>>
>>
>> Cheers,
>> Richard.
>>
>>
>>
>>
>> On 07/02/2012 16:30, JD Wong wrote:
>>
>>> Sure
>>>
>>> On Tue, Feb 7, 2012 at 8:34 AM, Richard Smith <richard at flymine.org
>>> <mailto:richard at flymine.org>> wrote:
>>>
>>>    JD,
>>>    Could you send us the intermine.log from your integrate directory
>>> after
>>>    running a build.  This is the most helpful thing for us to investigate
>>>    performance.
>>>
>>>    Thanks,
>>>    Richard.
>>>
>>>
>>>
>>>
>>>
>>>    On 06/02/2012 19:01, JD Wong wrote:
>>>
>>>        I was wondering how the other mods speed up their builds.  I have
>>>        configured ant, java, and postgres accordingly without effect.
>>>          I was
>>>        hoping to get the community's advice on this aspect.
>>>
>>>        Cheers,
>>>        -JD
>>>
>>>        On Thu, Feb 2, 2012 at 10:11 AM, JD Wong <jdmswong at gmail.com
>>>        <mailto:jdmswong at gmail.com>
>>>        <mailto:jdmswong at gmail.com <mailto:jdmswong at gmail.com>>> wrote:
>>>
>>>            I haven't given ant and postgres enough to consume all the
>>>        memory
>>>            when running simultaneously.  Also there is plenty of free
>>>        memory
>>>            during loading.
>>>
>>>            -JD
>>>
>>>
>>>            On Wed, Feb 1, 2012 at 10:35 PM, Josh Goodman
>>>        <jogoodma at indiana.edu <mailto:jogoodma at indiana.edu>
>>>        <mailto:jogoodma at indiana.edu <mailto:jogoodma at indiana.edu>>**>
>>> wrote:
>>>
>>>                You need to be careful of the various memory settings
>>>        here.  If you
>>>                set ant really high (>25% of total memory) and you are
>>> also
>>>                setting Pg
>>>                high you could be suffering from the two of them
>>>        fighting over
>>>                system
>>>                resources and causing the swap to get thrashed.  I would
>>>        run the
>>>                unix
>>>        "free" command while you are running a load to see what is going
>>> on
>>>                with memory.
>>>
>>>                e.g.
>>>
>>>                free -m -s 5
>>>
>>>                If you have other processes running on this machine
>>> (tomcat
>>>                instances)
>>>                you also need to adjust ant and Pg to take that into
>>>        account.
>>>
>>>                Josh
>>>
>>>                On Wed, Feb 1, 2012 at 5:20 PM, JD Wong
>>>        <jdmswong at gmail.com <mailto:jdmswong at gmail.com>
>>>        <mailto:jdmswong at gmail.com <mailto:jdmswong at gmail.com>>> wrote:
>>>         > Hi all,
>>>         > Loading my Go source takes on average 2500 seconds.  I have
>>>                tuned the
>>>         > postgres configuration paramaters to the desired values and
>>>                gave ant high
>>>         > heap memory to no avail.  Is there a way to speed this up?
>>>         >
>>>         > -JD
>>>         >
>>>         > ______________________________**___________________
>>>
>>>         > dev mailing list
>>>         > dev at intermine.org <mailto:dev at intermine.org>
>>>        <mailto:dev at intermine.org <mailto:dev at intermine.org>>
>>>         > http://mail.intermine.org/cgi-**__bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-__bin/mailman/listinfo/dev>
>>>        <http://mail.intermine.org/**cgi-bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-bin/mailman/listinfo/dev>
>>> >
>>>
>>>         >
>>>
>>>
>>>
>>>
>>>
>>>        ______________________________**___________________
>>>
>>>        dev mailing list
>>>        dev at intermine.org <mailto:dev at intermine.org>
>>>        http://mail.intermine.org/cgi-**__bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-__bin/mailman/listinfo/dev>
>>>        <http://mail.intermine.org/**cgi-bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-bin/mailman/listinfo/dev>
>>> >
>>>
>>>
>>>
>>>    ______________________________**___________________
>>>
>>>    dev mailing list
>>>    dev at intermine.org <mailto:dev at intermine.org>
>>>    http://mail.intermine.org/cgi-**__bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-__bin/mailman/listinfo/dev>
>>>    <http://mail.intermine.org/**cgi-bin/mailman/listinfo/dev<http://mail.intermine.org/cgi-bin/mailman/listinfo/dev>
>>> >
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.intermine.org/pipermail/dev/attachments/20120210/eab2d88f/attachment.html>


More information about the dev mailing list