[InterMine Dev] About the ObjectStoreQueryDurationException

Fengyuan Hu fh293 at cam.ac.uk
Fri Mar 15 11:45:45 GMT 2013


Hi Chen,

Thanks for your keen observation. You are very right about the double 
setting of "os.query.max-time", "os.query.max-limit" and 
"os.query.max-offset" in intermine.properties (Yes, I mean the one in 
your ~/.intermine folder, it will be renamed to intermine.properties 
whilst the webapp release) and default.intermine.webapp.properties 
(renamed to default.intermine.properties).

Looking into PropertiesUtil.java source code, you can find that it will 
parse the properties from, firstly default.intermine.properties and then 
intermine.properties. If we set those paras in two places, the one in 
intermine.properties will overwrite the one in 
default.intermine.properties. I don't know at what point we started to 
configure those paras in intermine.properties (perhaps not in your 
case), I assume the properties in intermine.properties are more often to 
change, we just put them there for our convenience, that's also why I 
suggested to add and edit max-time in that file. but I do think to keep 
them in one place (in default.intermine.properties) only is a good 
practice.

Let's now back to your question.
a) Could you log the max-time in ObjectStoreAbstractImpl class. You can 
log it in the constructor, right after the line:

    maxTime = Long.parseLong((String) props.get("max-time"));

Notice that there is a bug here, if you don't set it in the properties 
file, max-time is null, it will throw a ClassCastException. I just fixed it.

b) I tested it yesterday. I set max-time to a small value, say 1000, it 
should basically filter out any big query. I run a query to search all 
genes in flymine, and the results page looked like:
results

Does it look familiar to you? When you say "got a blank result", is this 
something like the screenshot? We believe this is a bug, it should give 
a query-too-long-to-run message. We've already created a ticket to fix 
it soon.

Cheers
Fengyuan

On 15/03/13 09:15, Chen Yian wrote:
> Hi Fengyuan,
>
> Do you mean the properties file at ~/.intermine folder ?
> I've tired to add the property and rebuilt my application but I still 
> got a blank result, after the "building table" message (when tried to 
> run a long query).
> By the way, I also noticed that the property os.query.max-time has 
> already set in the "default.intermine.webapp.properties" file.
> Did I misunderstand anything?
>
> Best,
>
> Chen
>
>
> (2013/03/15 1:58), Fengyuan Hu wrote:
>> Hi Chen,
>>
>> You can tweak "os.query.max-time" in MINE.properties to filter out 
>> long queries (e.g. os.query.max-time=10000000). The webapp should be 
>> able to catch it and show a warning. Could you give a try?
>>
>> Cheers
>> Fengyuan
>>
>> On 13/03/13 01:54, Chen Yian wrote:
>>> Hi Fengyuan,
>>>
>>> Thanks for your reply.
>>> I'm sorry I forgot to attach the query, it is as follow:
>>>
>>> <query model="genomic" 
>>> view="Gene.goAnnotation.subject.primaryIdentifier 
>>> Gene.goAnnotation.subject.name Gene.goAnnotation.subject.symbol
>>> Gene.goAnnotation.ontologyTerm.identifier 
>>> Gene.goAnnotation.ontologyTerm.name Gene.pathways.identifier 
>>> Gene.pathways.name
>>> Gene.proteins.primaryIdentifier Gene.proteins.primaryAccession 
>>> Gene.proteins.name Gene.proteins.organism.name
>>> Gene.proteins.proteinDomainRegions.proteinDomain.primaryIdentifier 
>>> Gene.proteins.proteinDomainRegions.proteinDomain.name
>>> Gene.proteins.proteinDomainRegions.proteinDomain.shortName 
>>> Gene.proteins.proteinDomainRegions.proteinDomain.type
>>> Gene.proteins.structuralDomains.cathClassification.level 
>>> Gene.proteins.structuralDomains.cathClassification.cathCode
>>> Gene.proteins.structuralDomains.cathClassification.description" 
>>> sortOrder="Gene.goAnnotation.subject.primaryIdentifier ASC" ></query>
>>>
>>> Though some classes or attributes are TargetMine specific, but I 
>>> think it may not be difficult to image how the query was.
>>>
>>> I understand most of  the case Postgres may take care the thread, 
>>> and close it,
>>> I just wonder if it is possible to execute something like 
>>> pg_cancel_backend when we throw an exception and decide to quit?
>>>
>>> By the way, I also think we should catch the exception, log the 
>>> query and response to the user with some warning.
>>> What do you think?
>>>
>>> All the best,
>>>
>>> Chen
>>>
>>> (2013/03/13 1:34), Fengyuan Hu wrote:
>>>> Hi Chen,
>>>>
>>>> Sorry you had the problem. Postgres will normally take some time to 
>>>> close the threads, for a large query, it's suppose to take longer, 
>>>> and it's bad at estimating running time of a query.
>>>> It's a bit hard for us to investigate further without accessing 
>>>> your mine from outside world. Could send us an example of the query?
>>>>
>>>> Cheers
>>>> Fengyuan
>>>>
>>>> On 12/03/13 09:22, Chen Yian wrote:
>>>>> Hi all,
>>>>>
>>>>> Someone constructed an improper query which associated with many 
>>>>> classes and without any constraint.
>>>>> Since the query could be expected to return several million results,
>>>>> when it was executed I got the ObjectStoreQueryDurationException 
>>>>> finally.
>>>>> Though the system threw the Exception and stopped working for the 
>>>>> query, PostgreSQL was still running.
>>>>> These threads occupied all resources for a while and some of them 
>>>>> even hang.
>>>>> Probably the problem itself is related to PostgreSQL,
>>>>> but is there any suggestion for preventing this kind of 'accident'?
>>>>>
>>>>> Best,
>>>>>
>>>>> Chen
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> dev mailing list
>>>>> dev at intermine.org
>>>>> http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
>>>>
>>>>
>>>
>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.intermine.org/pipermail/dev/attachments/20130315/ab22e4b2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FlyMine: Query results page.png
Type: image/png
Size: 23889 bytes
Desc: not available
URL: <http://mail.intermine.org/pipermail/dev/attachments/20130315/ab22e4b2/attachment-0001.png>


More information about the dev mailing list