[InterMine Dev] About the ObjectStoreQueryDurationException

Chen Yian chenyian at nibio.go.jp
Mon Mar 18 03:09:50 GMT 2013


Hi Fengyuan,

Thanks for your explanation. It's very clear.
I've tried to set a short max-time (1000ms) , and I was able to 
reproduce the exception easily now.
I also tried logging the max-time, it was exactly same as what I set in 
MINE.properties file (the default was overwritten).

What I meant blank result is almost as same as your screen shot, except 
I didn't get the "Oops!" warning at the feedback message area.
Any suggestion about where should I look into?

Best,

Chen


(2013/03/15 20:45), Fengyuan Hu wrote:
> Hi Chen,
>
> Thanks for your keen observation. You are very right about the double 
> setting of "os.query.max-time", "os.query.max-limit" and 
> "os.query.max-offset" in intermine.properties (Yes, I mean the one in 
> your ~/.intermine folder, it will be renamed to intermine.properties 
> whilst the webapp release) and default.intermine.webapp.properties 
> (renamed to default.intermine.properties).
>
> Looking into PropertiesUtil.java source code, you can find that it 
> will parse the properties from, firstly default.intermine.properties 
> and then intermine.properties. If we set those paras in two places, 
> the one in intermine.properties will overwrite the one in 
> default.intermine.properties. I don't know at what point we started to 
> configure those paras in intermine.properties (perhaps not in your 
> case), I assume the properties in intermine.properties are more often 
> to change, we just put them there for our convenience, that's also why 
> I suggested to add and edit max-time in that file. but I do think to 
> keep them in one place (in default.intermine.properties) only is a 
> good practice.
>
> Let's now back to your question.
> a) Could you log the max-time in ObjectStoreAbstractImpl class. You 
> can log it in the constructor, right after the line:
>
>     maxTime = Long.parseLong((String) props.get("max-time"));
>
> Notice that there is a bug here, if you don't set it in the properties 
> file, max-time is null, it will throw a ClassCastException. I just 
> fixed it.
>
> b) I tested it yesterday. I set max-time to a small value, say 1000, 
> it should basically filter out any big query. I run a query to search 
> all genes in flymine, and the results page looked like:
> results
>
> Does it look familiar to you? When you say "got a blank result", is 
> this something like the screenshot? We believe this is a bug, it 
> should give a query-too-long-to-run message. We've already created a 
> ticket to fix it soon.
>
> Cheers
> Fengyuan
>
> On 15/03/13 09:15, Chen Yian wrote:
>> Hi Fengyuan,
>>
>> Do you mean the properties file at ~/.intermine folder ?
>> I've tired to add the property and rebuilt my application but I still 
>> got a blank result, after the "building table" message (when tried to 
>> run a long query).
>> By the way, I also noticed that the property os.query.max-time has 
>> already set in the "default.intermine.webapp.properties" file.
>> Did I misunderstand anything?
>>
>> Best,
>>
>> Chen
>>
>>
>> (2013/03/15 1:58), Fengyuan Hu wrote:
>>> Hi Chen,
>>>
>>> You can tweak "os.query.max-time" in MINE.properties to filter out 
>>> long queries (e.g. os.query.max-time=10000000). The webapp should be 
>>> able to catch it and show a warning. Could you give a try?
>>>
>>> Cheers
>>> Fengyuan
>>>
>>> On 13/03/13 01:54, Chen Yian wrote:
>>>> Hi Fengyuan,
>>>>
>>>> Thanks for your reply.
>>>> I'm sorry I forgot to attach the query, it is as follow:
>>>>
>>>> <query model="genomic" 
>>>> view="Gene.goAnnotation.subject.primaryIdentifier 
>>>> Gene.goAnnotation.subject.name Gene.goAnnotation.subject.symbol
>>>> Gene.goAnnotation.ontologyTerm.identifier 
>>>> Gene.goAnnotation.ontologyTerm.name Gene.pathways.identifier 
>>>> Gene.pathways.name
>>>> Gene.proteins.primaryIdentifier Gene.proteins.primaryAccession 
>>>> Gene.proteins.name Gene.proteins.organism.name
>>>> Gene.proteins.proteinDomainRegions.proteinDomain.primaryIdentifier 
>>>> Gene.proteins.proteinDomainRegions.proteinDomain.name
>>>> Gene.proteins.proteinDomainRegions.proteinDomain.shortName 
>>>> Gene.proteins.proteinDomainRegions.proteinDomain.type
>>>> Gene.proteins.structuralDomains.cathClassification.level 
>>>> Gene.proteins.structuralDomains.cathClassification.cathCode
>>>> Gene.proteins.structuralDomains.cathClassification.description" 
>>>> sortOrder="Gene.goAnnotation.subject.primaryIdentifier ASC" ></query>
>>>>
>>>> Though some classes or attributes are TargetMine specific, but I 
>>>> think it may not be difficult to image how the query was.
>>>>
>>>> I understand most of  the case Postgres may take care the thread, 
>>>> and close it,
>>>> I just wonder if it is possible to execute something like 
>>>> pg_cancel_backend when we throw an exception and decide to quit?
>>>>
>>>> By the way, I also think we should catch the exception, log the 
>>>> query and response to the user with some warning.
>>>> What do you think?
>>>>
>>>> All the best,
>>>>
>>>> Chen
>>>>
>>>> (2013/03/13 1:34), Fengyuan Hu wrote:
>>>>> Hi Chen,
>>>>>
>>>>> Sorry you had the problem. Postgres will normally take some time 
>>>>> to close the threads, for a large query, it's suppose to take 
>>>>> longer, and it's bad at estimating running time of a query.
>>>>> It's a bit hard for us to investigate further without accessing 
>>>>> your mine from outside world. Could send us an example of the query?
>>>>>
>>>>> Cheers
>>>>> Fengyuan
>>>>>
>>>>> On 12/03/13 09:22, Chen Yian wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> Someone constructed an improper query which associated with many 
>>>>>> classes and without any constraint.
>>>>>> Since the query could be expected to return several million results,
>>>>>> when it was executed I got the ObjectStoreQueryDurationException 
>>>>>> finally.
>>>>>> Though the system threw the Exception and stopped working for the 
>>>>>> query, PostgreSQL was still running.
>>>>>> These threads occupied all resources for a while and some of them 
>>>>>> even hang.
>>>>>> Probably the problem itself is related to PostgreSQL,
>>>>>> but is there any suggestion for preventing this kind of 'accident'?
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Chen
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> dev mailing list
>>>>>> dev at intermine.org
>>>>>> http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.intermine.org/pipermail/dev/attachments/20130318/b2a05f33/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 23889 bytes
Desc: not available
URL: <http://mail.intermine.org/pipermail/dev/attachments/20130318/b2a05f33/attachment-0001.png>


More information about the dev mailing list