[InterMine Dev] errors in build a new mine

Julie Sullivan julie at flymine.org
Tue Oct 4 08:57:35 BST 2011


Hi Su,

I think this is the relevant part of the error message:

 > Caused by: org.postgresql.util.PSQLException: FATAL: sorry, too many clients 
already

You are running out of database connections.  I would try to use the 
project_build script instead of running `ant -Dsource=all`.  See here for 
documentation of how to use the script:

	http://intermine.org/wiki/GettingStarted#a12BuildingaMine

Using `ant -Dsource=all` runs the entire integration in one huge ant task. 
Unfortunately, ant isolates each individual task in its own ClassLoader, which 
prevents the different tasks from sharing database connections. Each ClassLoader 
will end up having its own connection pool, with a maximum number of connections 
as set in the properties. Moreover, it doesn't tend to allow those ClassLoaders 
or database connection pools to be garbage collected once the task has finished. 
Therefore, each individual task will leave several idle database connections 
lying around. We tried to work around this problem a while back too, but were 
unsuccessful. Ant simply isolates the individual tasks too well.

For this reason, we wrote a script that would perform the integration for you by 
executing ant once per data source. It also does things like make backups of the 
database to allow restarting from half-way through. It is in 
bio/scripts/project_build. We recommend you use this rather than performing the 
whole build in a single ant.

Cheers,
Julie

On 03/10/11 16:22, wang su wrote:
> hi ,
> thanks for your help,
> but I get another error like
>
> BUILD FAILED
> /mnt/Storage/home/wangs/intermine_0_97/imbuild/integrate.xml:54: The following error occurred while executing this line:
> /mnt/Storage/home/wangs/intermine_0_97/imbuild/source.xml:201: Exception in XmlDataLoaderTask
> 	at org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:168)
> 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:616)
> 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
> 	at org.apache.tools.ant.Task.perform(Task.java:348)
> 	at org.apache.tools.ant.Target.execute(Target.java:390)
> 	at org.apache.tools.ant.Target.performTasks(Target.java:411)
> 	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1360)
> 	at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
> 	at org.apache.tools.ant.Project.executeTargets(Project.java:1212)
> 	at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:441)
> 	at org.intermine.task.Integrate.performAction(Integrate.java:223)
> 	at org.intermine.task.Integrate.performAction(Integrate.java:135)
> 	at org.intermine.task.Integrate.execute(Integrate.java:127)
> 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:616)
> 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
> 	at org.apache.tools.ant.Task.perform(Task.java:348)
> 	at org.apache.tools.ant.Target.execute(Target.java:390)
> 	at org.apache.tools.ant.Target.performTasks(Target.java:411)
> 	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1360)
> 	at org.apache.tools.ant.Project.executeTarget(Project.java:1329)
> 	at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
> 	at org.apache.tools.ant.Project.executeTargets(Project.java:1212)
> 	at org.apache.tools.ant.Main.runBuild(Main.java:801)
> 	at org.apache.tools.ant.Main.startAnt(Main.java:218)
> 	at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
> 	at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
> Caused by: org.intermine.objectstore.ObjectStoreException: Failed to instantiate IntegrationWriter class: org.intermine.dataloader.IntegrationWriterDataTrackingImpl
> 	at org.intermine.dataloader.IntegrationWriterFactory.getIntegrationWriter(IntegrationWriterFactory.java:75)
> 	at org.intermine.dataloader.XmlDataLoaderTask.execute(XmlDataLoaderTask.java:123)
> 	... 31 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:616)
> 	at org.intermine.dataloader.IntegrationWriterFactory.getIntegrationWriter(IntegrationWriterFactory.java:65)
> 	... 32 more
> Caused by: java.lang.IllegalArgumentException: Problem instantiating IntegrationWriterDataTrackingImpl integration.production
> 	at org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:145)
> 	at org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:80)
> 	... 37 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> 	at org.intermine.dataloader.IntegrationWriterDataTrackingImpl.getInstance(IntegrationWriterDataTrackingImpl.java:137)
> 	... 38 more
> Caused by: java.lang.RuntimeException: can't open datasource for {platform=PostgreSQL, datasource.dataSourceName=db.production, datasource.maxConnections=50, datasource.databaseName=pathmine, datasource.class=org.postgresql.ds.PGPoolingDataSource, datasource.password=learningperl, datasource.user=postgres, driver=org.postgresql.Driver, datasource.serverName=localhost} org.postgresql.Driver PostgreSQL
> 	at org.intermine.sql.Database.getConnection(Database.java:106)
> 	at org.intermine.dataloader.DataTracker.<init>(DataTracker.java:91)
> 	... 43 more
> Caused by: org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
> 	at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:464)
> 	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112)
> 	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
> 	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
> 	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
> 	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
> 	at org.postgresql.jdbc3g.Jdbc3gConnection.<init>(Jdbc3gConnection.java:24)
> 	at org.postgresql.Driver.makeConnection(Driver.java:393)
> 	at org.postgresql.Driver.connect(Driver.java:267)
> 	at java.sql.DriverManager.getConnection(DriverManager.java:620)
> 	at java.sql.DriverManager.getConnection(DriverManager.java:200)
> 	at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:91)
> 	at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:74)
> 	at org.postgresql.ds.PGConnectionPoolDataSource.getPooledConnection(PGConnectionPoolDataSource.java:60)
> 	at org.postgresql.ds.jdbc23.AbstractJdbc23PoolingDataSource.getPooledConnection(AbstractJdbc23PoolingDataSource.java:429)
> 	at org.postgresql.ds.jdbc23.AbstractJdbc23PoolingDataSource.getConnection(AbstractJdbc23PoolingDataSource.java:356)
> 	at org.intermine.sql.Database.getConnection(Database.java:104)
> 	... 44 more
>
> Total time: 9 minutes 11 seconds
>
> Thanks
>
> Su
> On 2011-10-3, at 下午7:00, dev-request at intermine.org wrote:
>
>> Send dev mailing list submissions to
>> 	dev at intermine.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> 	http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
>> or, via email, send a message with subject or body 'help' to
>> 	dev-request at intermine.org
>>
>> You can reach the person managing the list at
>> 	dev-owner at intermine.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of dev digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: Changing the default summary views (Julie Sullivan)
>>    2. Re: Reports from 'Usage' Tab Emailable? (Julie Sullivan)
>>    3. Re: constraints editable by default (Julie Sullivan)
>>    4. errors in build a new mine (wang su)
>>    5. Re: errors in build a new mine (Julie Sullivan)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 03 Oct 2011 08:43:29 +0100
>> From: Julie Sullivan<julie at flymine.org>
>> To: dev at intermine.org
>> Subject: Re: [InterMine Dev] Changing the default summary views
>> Message-ID:<4E8967A1.3050305 at flymine.org>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> That is controlled by the webconfig-model.xml file.
>>
>> Here's a list of properties files and what they do:
>>
>> 	http://intermine.org/wiki/WebappConfig
>>
>>
>> On 30/09/11 20:23, JD Wong wrote:
>>> Hi,
>>>
>>> Does anybody know how to change what data is displayed when viewing a
>>> record?
>>>
>>> Thanks!
>>> -JD
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> dev mailing list
>>> dev at intermine.org
>>> http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Mon, 03 Oct 2011 08:59:02 +0100
>> From: Julie Sullivan<julie at flymine.org>
>> To: dev at intermine.org
>> Subject: Re: [InterMine Dev] Reports from 'Usage' Tab Emailable?
>> Message-ID:<4E896B46.3020705 at flymine.org>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> No, they aren't, but that's a really good idea.  I've made a ticket.
>>
>> On 30/09/11 22:47, Kalpana Karra wrote:
>>> Wondering if that is a possibility and I am just not seeing it ;)
>>> Thanks,
>>> -Kalpana
>>>
>>> _______________________________________________
>>> dev mailing list
>>> dev at intermine.org
>>> http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
>>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Mon, 03 Oct 2011 09:16:33 +0100
>> From: Julie Sullivan<julie at flymine.org>
>> To: dev at intermine.org
>> Subject: Re: [InterMine Dev] constraints editable by default
>> Message-ID:<4E896F61.7080605 at flymine.org>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Can you go into more detail as to how you are getting a "new" template from an
>> old one?  There are a couple ways to do it.
>>
>> If you edit a current template, all "editable" options should remain the same
>> unless changed by you.
>>
>> If you make a new template, all "editable" options are by default unlocked.
>>
>> How are you making a "new" template?  Renaming?  Because that won't make a new
>> template.  Export/import XML?  I just tested and the editable values were copied
>> over successfully - in FlyMine and on our InterMine 0.98 beta site.
>>
>> If you click on "edit query" on the template form to get to the QueryBuilder,
>> then you are editing the query not the template.  All template-related
>> information for that query is lost.  When the template constraints are re-added,
>> they default to be editable.  I would try exporting/importing the XML instead.
>>
>> If none of that works for you, upgrade to 0.98 and try again.  It must have been
>> a bug that we've already fixed.  Let me know if that doesn't fix the problem!
>>
>> On 30/09/11 22:16, Kalpana Karra wrote:
>>> Hi,
>>>
>>> Not sure if it was always like this or was changed intentionally in one of the past releases..
>>> but..I am finding that if I take an existing template,edit it and start building a new template all
>>> the constraints are by default unlocked.
>>>
>>>   I have to go through each one and uncheck 'Editable' option. (in YM we do have quite a few  non-editable
>>
>> constraints built into most templates)
>>>
>>> Could this be changed such that everything remains the same as the template that one starts working off of??
>>>
>>> Thanks,
>>> -Kalpana
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> dev mailing list
>>> dev at intermine.org
>>> http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
>>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Mon, 3 Oct 2011 16:15:04 +0800
>> From: wang su<wangsu0623 at gmail.com>
>> To: dev at intermine.org
>> Subject: [InterMine Dev] errors in build a new mine
>> Message-ID:<1AF98BD0-136F-47CA-B530-7C049C06DB06 at gmail.com>
>> Content-Type: text/plain; charset=us-ascii
>>
>> hi ,
>> I am loading data into my database
>> # in MINE_NAME/integrate
>> ant -v -Dsource=all
>> then, there are some errors happened ,before building  this mine, I had  built the malariamine following the start steps. The error is like:
>>
>>
>>
>> BUILD FAILED
>> /mnt/Storage/home/wangs/intermine_0_97/imbuild/integrate.xml:54: The following error occurred while executing this line:
>> /mnt/Storage/home/wangs/intermine_0_97/imbuild/source.xml:330: java.lang.RuntimeException: Exception while dataloading - to allow multiple errors, set the property "dataLoader.allowMultipleErrors" to true
>> Problem while loading item identifier 1_1 because
>> There is already an equivalent in the database from this source (<Source: name="uniprot-path", type="uniprot", skeleton=false>) from a *previous* run; object from source in this run: "DataSource [description="null", id="4", name="UniProt", url="null"]", object from database: "DataSource [description="null", id="1000001", name="UniProt", url="null"]"; noticed problem while merging field "name" originally read from source:<Source: name="uniprot-path", type="uniprot", skeleton=false>
>> 	at org.intermine.dataloader.ObjectStoreDataLoader.process(ObjectStoreDataLoader.java:164)
>> 	at org.intermine.dataloader.ObjectStoreDataLoader.process(ObjectStoreDataLoader.java:60)
>> 	at org.intermine.dataloader.ObjectStoreDataLoaderTask.execute(ObjectStoreDataLoaderTask.java:128)
>> 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
>> 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> 	at java.lang.reflect.Method.invoke(Method.java:616)
>> 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>> 	at org.apache.tools.ant.Task.perform(Task.java:348)
>> 	at org.apache.tools.ant.Target.execute(Target.java:390)
>> 	at org.apache.tools.ant.Target.performTasks(Target.java:411)
>> 	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1360)
>> 	at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
>> 	at org.apache.tools.ant.Project.executeTargets(Project.java:1212)
>> 	at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:441)
>> 	at org.intermine.task.Integrate.performAction(Integrate.java:223)
>> 	at org.intermine.task.Integrate.performAction(Integrate.java:136)
>> 	at org.intermine.task.Integrate.execute(Integrate.java:127)
>> 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
>> 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> 	at java.lang.reflect.Method.invoke(Method.java:616)
>> 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>> 	at org.apache.tools.ant.Task.perform(Task.java:348)
>> 	at org.apache.tools.ant.Target.execute(Target.java:390)
>> 	at org.apache.tools.ant.Target.performTasks(Target.java:411)
>> 	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1360)
>> 	at org.apache.tools.ant.Project.executeTarget(Project.java:1329)
>> 	at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
>> 	at org.apache.tools.ant.Project.executeTargets(Project.java:1212)
>> 	at org.apache.tools.ant.Main.runBuild(Main.java:801)
>> 	at org.apache.tools.ant.Main.startAnt(Main.java:218)
>> 	at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
>> 	at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
>> Caused by: java.lang.IllegalArgumentException: There is already an equivalent in the database from this source (<Source: name="uniprot-path", type="uniprot", skeleton=false>) from a *previous* run; object from source in this run: "DataSource [description="null", id="4", name="UniProt", url="null"]", object from database: "DataSource [description="null", id="1000001", name="UniProt", url="null"]"; noticed problem while merging field "name" originally read from source:<Source: name="uniprot-path", type="uniprot", skeleton=false>
>> 	at org.intermine.dataloader.IntegrationWriterDataTrackingImpl.store(IntegrationWriterDataTrackingImpl.java:372)
>> 	at org.intermine.dataloader.IntegrationWriterAbstractImpl.store(IntegrationWriterAbstractImpl.java:171)
>> 	at org.intermine.dataloader.ObjectStoreDataLoader.process(ObjectStoreDataLoader.java:146)
>> 	... 33 more
>>
>> Total time: 1 minute 4 seconds
>>
>>
>>
>> Thanks
>> Su
>> Tongji University , Shanghai China
>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Mon, 03 Oct 2011 09:25:01 +0100
>> From: Julie Sullivan<julie at flymine.org>
>> To: dev at intermine.org
>> Subject: Re: [InterMine Dev] errors in build a new mine
>> Message-ID:<4E89715D.8090108 at flymine.org>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Hello Su!
>>
>> "Caused by: java.lang.IllegalArgumentException: There is already an equivalent
>> in the database from this source "
>>
>> This error message means that you've run the same source twice, UniProt.  Maybe
>> you didn't run the "build-db" task first?
>>
>> Try this:
>>
>>   1. ant -v build-db
>>   2. ant -v -Dsource=all
>>
>> Let me know if that doesn't work for you.
>>
>> Cheers
>> Julie
>>
>> On 03/10/11 09:15, wang su wrote:
>>> Caused by: java.lang.IllegalArgumentException: There is already an equivalent in the database from this
>>
>> source (<Source: name="uniprot-path", type="uniprot", skeleton=false>) from
>> a*previous*  run; object
>>
>> from source in this run: "DataSource [description="null", id="4",
>> name="UniProt", url="null"]", object
>>
>> from database: "DataSource [description="null", id="1000001", name="UniProt",
>> url="null"]"; noticed problem
>>
>> while merging field "name" originally read from source:<Source:
>> name="uniprot-path", type="uniprot", skeleton=false>
>>
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> dev mailing list
>> dev at intermine.org
>> http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
>>
>>
>> End of dev Digest, Vol 29, Issue 2
>> **********************************
>
>
> _______________________________________________
> dev mailing list
> dev at intermine.org
> http://mail.intermine.org/cgi-bin/mailman/listinfo/dev



More information about the dev mailing list