[InterMine Dev] About the ObjectStoreQueryDurationException

Fengyuan Hu fh293 at cam.ac.uk
Mon Mar 18 15:28:25 GMT 2013


Hi Chen,

The message should be embedded in imtabls.js. An older version of this 
script is here - intermine/webapp/main/resources/webapp/js/imtables.js 
<https://github.com/intermine/intermine/blob/dev/intermine/webapp/main/resources/webapp/js/imtables.js>

However, we started to host all the static files (js and css) on CDN 
<https://github.com/intermine/CDN> recently (a separate repo, the older 
scripts will be overwritten). If your mine is up-to-date, you can open a 
page "showProperties.do" in your browser as superuser, then search for 
cdn on the page. It points to the intermine CDN 
<http://cdn.intermine.org/> by default (or directly search 
"head.cdn.location" in global.web.properties). Newer version of 
imtabls.js 
<https://github.com/intermine/CDN/blob/master/js/intermine/im-tables/latest/imtables.js> 
is hosted there. Of course, you can host a local CDN if you'd like to 
either customise the scripts or simply be independent from us.

If you are still on an old branch which cdn is not configured in 
global.web.properties,  you can find the script where I pointed to you 
at the beginning.

Cheers
Fengyuan

On 18/03/13 03:09, Chen Yian wrote:
> Hi Fengyuan,
>
> Thanks for your explanation. It's very clear.
> I've tried to set a short max-time (1000ms) , and I was able to 
> reproduce the exception easily now.
> I also tried logging the max-time, it was exactly same as what I set 
> in MINE.properties file (the default was overwritten).
>
> What I meant blank result is almost as same as your screen shot, 
> except I didn't get the "Oops!" warning at the feedback message area.
> Any suggestion about where should I look into?
>
> Best,
>
> Chen
>
>
> (2013/03/15 20:45), Fengyuan Hu wrote:
>> Hi Chen,
>>
>> Thanks for your keen observation. You are very right about the double 
>> setting of "os.query.max-time", "os.query.max-limit" and 
>> "os.query.max-offset" in intermine.properties (Yes, I mean the one in 
>> your ~/.intermine folder, it will be renamed to intermine.properties 
>> whilst the webapp release) and default.intermine.webapp.properties 
>> (renamed to default.intermine.properties).
>>
>> Looking into PropertiesUtil.java source code, you can find that it 
>> will parse the properties from, firstly default.intermine.properties 
>> and then intermine.properties. If we set those paras in two places, 
>> the one in intermine.properties will overwrite the one in 
>> default.intermine.properties. I don't know at what point we started 
>> to configure those paras in intermine.properties (perhaps not in your 
>> case), I assume the properties in intermine.properties are more often 
>> to change, we just put them there for our convenience, that's also 
>> why I suggested to add and edit max-time in that file. but I do think 
>> to keep them in one place (in default.intermine.properties) only is a 
>> good practice.
>>
>> Let's now back to your question.
>> a) Could you log the max-time in ObjectStoreAbstractImpl class. You 
>> can log it in the constructor, right after the line:
>>
>>     maxTime = Long.parseLong((String) props.get("max-time"));
>>
>> Notice that there is a bug here, if you don't set it in the 
>> properties file, max-time is null, it will throw a 
>> ClassCastException. I just fixed it.
>>
>> b) I tested it yesterday. I set max-time to a small value, say 1000, 
>> it should basically filter out any big query. I run a query to search 
>> all genes in flymine, and the results page looked like:
>> results
>>
>> Does it look familiar to you? When you say "got a blank result", is 
>> this something like the screenshot? We believe this is a bug, it 
>> should give a query-too-long-to-run message. We've already created a 
>> ticket to fix it soon.
>>
>> Cheers
>> Fengyuan
>>
>> On 15/03/13 09:15, Chen Yian wrote:
>>> Hi Fengyuan,
>>>
>>> Do you mean the properties file at ~/.intermine folder ?
>>> I've tired to add the property and rebuilt my application but I 
>>> still got a blank result, after the "building table" message (when 
>>> tried to run a long query).
>>> By the way, I also noticed that the property os.query.max-time has 
>>> already set in the "default.intermine.webapp.properties" file.
>>> Did I misunderstand anything?
>>>
>>> Best,
>>>
>>> Chen
>>>
>>>
>>> (2013/03/15 1:58), Fengyuan Hu wrote:
>>>> Hi Chen,
>>>>
>>>> You can tweak "os.query.max-time" in MINE.properties to filter out 
>>>> long queries (e.g. os.query.max-time=10000000). The webapp should 
>>>> be able to catch it and show a warning. Could you give a try?
>>>>
>>>> Cheers
>>>> Fengyuan
>>>>
>>>> On 13/03/13 01:54, Chen Yian wrote:
>>>>> Hi Fengyuan,
>>>>>
>>>>> Thanks for your reply.
>>>>> I'm sorry I forgot to attach the query, it is as follow:
>>>>>
>>>>> <query model="genomic" 
>>>>> view="Gene.goAnnotation.subject.primaryIdentifier 
>>>>> Gene.goAnnotation.subject.name Gene.goAnnotation.subject.symbol
>>>>> Gene.goAnnotation.ontologyTerm.identifier 
>>>>> Gene.goAnnotation.ontologyTerm.name Gene.pathways.identifier 
>>>>> Gene.pathways.name
>>>>> Gene.proteins.primaryIdentifier Gene.proteins.primaryAccession 
>>>>> Gene.proteins.name Gene.proteins.organism.name
>>>>> Gene.proteins.proteinDomainRegions.proteinDomain.primaryIdentifier 
>>>>> Gene.proteins.proteinDomainRegions.proteinDomain.name
>>>>> Gene.proteins.proteinDomainRegions.proteinDomain.shortName 
>>>>> Gene.proteins.proteinDomainRegions.proteinDomain.type
>>>>> Gene.proteins.structuralDomains.cathClassification.level 
>>>>> Gene.proteins.structuralDomains.cathClassification.cathCode
>>>>> Gene.proteins.structuralDomains.cathClassification.description" 
>>>>> sortOrder="Gene.goAnnotation.subject.primaryIdentifier ASC" ></query>
>>>>>
>>>>> Though some classes or attributes are TargetMine specific, but I 
>>>>> think it may not be difficult to image how the query was.
>>>>>
>>>>> I understand most of  the case Postgres may take care the thread, 
>>>>> and close it,
>>>>> I just wonder if it is possible to execute something like 
>>>>> pg_cancel_backend when we throw an exception and decide to quit?
>>>>>
>>>>> By the way, I also think we should catch the exception, log the 
>>>>> query and response to the user with some warning.
>>>>> What do you think?
>>>>>
>>>>> All the best,
>>>>>
>>>>> Chen
>>>>>
>>>>> (2013/03/13 1:34), Fengyuan Hu wrote:
>>>>>> Hi Chen,
>>>>>>
>>>>>> Sorry you had the problem. Postgres will normally take some time 
>>>>>> to close the threads, for a large query, it's suppose to take 
>>>>>> longer, and it's bad at estimating running time of a query.
>>>>>> It's a bit hard for us to investigate further without accessing 
>>>>>> your mine from outside world. Could send us an example of the query?
>>>>>>
>>>>>> Cheers
>>>>>> Fengyuan
>>>>>>
>>>>>> On 12/03/13 09:22, Chen Yian wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Someone constructed an improper query which associated with many 
>>>>>>> classes and without any constraint.
>>>>>>> Since the query could be expected to return several million 
>>>>>>> results,
>>>>>>> when it was executed I got the ObjectStoreQueryDurationException 
>>>>>>> finally.
>>>>>>> Though the system threw the Exception and stopped working for 
>>>>>>> the query, PostgreSQL was still running.
>>>>>>> These threads occupied all resources for a while and some of 
>>>>>>> them even hang.
>>>>>>> Probably the problem itself is related to PostgreSQL,
>>>>>>> but is there any suggestion for preventing this kind of 'accident'?
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Chen
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> dev mailing list
>>>>>>> dev at intermine.org
>>>>>>> http://mail.intermine.org/cgi-bin/mailman/listinfo/dev
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.intermine.org/pipermail/dev/attachments/20130318/6155a28b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 23889 bytes
Desc: not available
URL: <http://mail.intermine.org/pipermail/dev/attachments/20130318/6155a28b/attachment-0001.png>


More information about the dev mailing list