[InterMine Dev] Using perl client library with https

Joe Carlson jwcarlson at lbl.gov
Thu Feb 18 06:05:54 GMT 2016


> On Feb 17, 2016, at 9:19 PM, Josh Goodman <jogoodma at indiana.edu> wrote:
> 
> Hi Joe,
> 
> Apologies for the incorrect syntax, I mixed up Moose vs MooseX types.  I am not a maintainer of the CPAN module, but I created a pull request so that one of the InterMine devs can take a look.
> 
> https://github.com/intermine/intermine/pull/1308 <https://github.com/intermine/intermine/pull/1308>
> 

yep. this was pretty much what I did, too.

Motivated by this issue, I checked the other API’s to see if they had any trouble with https. Java and python passed my “exhaustive" testing (1 script each).

But Ruby failed. When I tried my first script, it sent me a nastygram:


> ruby template.rb 
/Library/Ruby/Gems/2.0.0/gems/intermine-1.04.00/lib/intermine/service.rb:173:in `rescue in initialize': Error fetching version at https://phytozome.jgi.doe.gov/phytomine/service/version: <https://phytozome.jgi.doe.gov/phytomine/service/version:> wrong status line: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">" (InterMine::ServiceError)
	from /Library/Ruby/Gems/2.0.0/gems/intermine-1.04.00/lib/intermine/service.rb:170:in `initialize'
	from template.rb:16:in `new'
	from template.rb:16:in `<main>'

I know absolutely no ruby. And I’m not even certain that this is a https issue. But it looks like it’s getting some unexpected html when requesting the version. Should this be checked by some rubyist?

Thanks for all your help,

Joe

> Cheers,
> Josh
> 
> On Wed, Feb 17, 2016 at 3:57 PM Joe Carlson <jwcarlson at lbl.gov <mailto:jwcarlson at lbl.gov>> wrote:
> Ah! I had added the class_type to Types.pm and I tried isa=>[NetHTTP,NetHTTPS]. That didn’t work. I didnt know about the pipe symbol.
> 
> But your syntax does not work for me with the quotes in there. I have more success if I write it as
> 
> isa => NetHTTP|NetHTTPS,
> 
> and it even work when I query flymine!
> 
> Are you folks the maintainers of the module in CPAN? Can you push out a update? 
> 
> thanks,
> 
> Joe
> 
>> On Feb 17, 2016, at 12:12 PM, Josh Goodman <jogoodma at indiana.edu <mailto:jogoodma at indiana.edu>> wrote:
>> 
>> Hi Joe,
>> 
>> You can use type unions for this.  Given how this has been implemented here, I would do the following.
>> 
>> Add a new type for NetHTTPS in Webservice/InterMine/Types.pm:
>> 
>> class_type NetHTTPS, { class => 'Net::HTTPS', };
>> 
>> Then modify ResultIterator to import the new type and use a type union on the connection attribute.
>> 
>> use Webservice::InterMine::Types qw(Uri HTTPCode NetHTTP NetHTTPS RowFormat JsonFormat RequestFormat RowParser);
>> ...
>> has connection => (
>>     is        => 'ro',
>>     isa       => 'NetHTTP|NetHTTPS',
>>     writer    => 'set_connection',
>>     predicate => 'has_connection',
>> );
>> 
>> Josh
>> 
>> On Wed, Feb 17, 2016 at 2:10 PM Joe Carlson <jwcarlson at lbl.gov <mailto:jwcarlson at lbl.gov>> wrote:
>> For what it’s worth, I’ve verified that if I add NetHTTPS as a type in Webservice::InterMine::Types, change the ‘isa’ in connection to NetHTTPS and open the connection with Net::HTTPS, then I can use the perl API over https.
>> 
>> But I don’t know enough about Moose to adapt the code to work with either Net::HTTP or Net::HTTPS. Line 520 in ResultIterator has a call $self->set_connection($connection) which checks that this is a specific class Net::HTTP. Is there a way to get Moose to accept alternative types in the type checking?
>> 
>> Joe
>> 
>> 
>>> On Feb 17, 2016, at 10:15 AM, Joe Carlson <jwcarlson at lbl.gov <mailto:jwcarlson at lbl.gov>> wrote:
>>> 
>>> New problem.
>>> 
>>> We have a mandate to use https exclusively for our web services. One of our users has seen that the perl client libraries do not work at our site. We have confirmed that this is a problem. There are some warning messages about deep recursion on subroutine "Webservice::InterMine::ResultIterator::set_headers” and the script never returns results.
>>> 
>>> I discovered the trick of setting the DEBUG environment variable to get some verbosity in the output. The first few fetches seem to work fine. There are some calls to fetch the service version and model. That goes well.Here is the output:
>>> 
>>>> FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version <https://phytozome.jgi.doe.gov/phytomine/service/version> 1455730008.88403 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
>>>> FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version <https://phytozome.jgi.doe.gov/phytomine/service/version> 1455730009.32112 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
>>>> FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml <https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml> 1455730009.32364 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 830.
>>>> FINISHED FETCHING https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml <https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml> 1455730011.81447 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line 839.
>>>> string=><model name="genomic" package="org.intermine.model.bio">
>>>> <class name="DatabaseReference" is-interface="true">
>>>>         <attribute name="identifier" type="java.lang.String"/>
>>>>         <reference name="subject" referenced-type="Publication" reverse-reference="crossReferences"/>
>>>>         <reference name="source" referenced-type="DataSource”/>
>>> ….
>>> 
>>> This went well and it appears to use the https connection retrieve the contents.
>>> 
>>> 
>>>> CONNECTING TO phytozome.jgi.doe.gov <http://phytozome.jgi.doe.gov/>:443 at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 493.
>>>> UNENCODED-PARAM: op1==
>>>> UNENCODED-PARAM: constraint1=Gene.primaryIdentifier
>>>> UNENCODED-PARAM: format=json
>>>> UNENCODED-PARAM: name=GO_terms_for_genes
>>>> UNENCODED-PARAM: value1=Potri.014G066100
>>>> UNENCODED-PARAM: code1=A
>>>> SENDING REQUESTPOST /phytomine/service/template/results HTTP/1.1
>>>> Connection: close
>>>> Host: phytozome.jgi.doe.gov <http://phytozome.jgi.doe.gov/>:443
>>>> Content-Type: application/x-www-form-urlencoded
>>>> User-Agent: Webservice::InterMine-1.0405/Perl client library
>>>> Content-Length: 110
>>>> 
>>>> op1=%3D&constraint1=Gene.primaryIdentifier&format=json&name=GO_terms_for_genes&value1=Potri.014G066100&code1=A
>>>> SETTING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 26.
>>>> RETURNING LINE: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 1.
>>>> RETURNING LINE: <html><head> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 2.
>>>> RETURNING LINE: <title>400 Bad Request</title> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 3.
>>>> RETURNING LINE: </head><body> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 4.
>>>> RETURNING LINE: <h1>Bad Request</h1> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 5.
>>>> RETURNING LINE: <p>Your browser sent a request that this server could not understand.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 6.
>>>> RETURNING LINE: Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 7.
>>>> HEADER Reason = You're speaking plain HTTP to an SSL-enabled server port.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 7.
>>>> RETURNING LINE: Instead use the HTTPS scheme to access this URL, please.<br /> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 8.
>>>> RETURNING LINE: <blockquote>Hint: <a href="https://gnweb01.nersc.gov/ <https://gnweb01.nersc.gov/>"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> <https://gnweb01.nersc.gov/%3C/b%3E%3C/a%3E%3C/blockquote%3E%3C/p%3E> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 9.
>>>> HEADER <blockquote>Hint = <a href="https://gnweb01.nersc.gov/ <https://gnweb01.nersc.gov/>"><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> <https://gnweb01.nersc.gov/%3C/b%3E%3C/a%3E%3C/blockquote%3E%3C/p%3E> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 308, <GEN3> line 9.
>>>> RETURNING LINE: </body></html> at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
>>>> RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
>>>> FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
>>>> RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
>>>> FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
>>>> RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
>>>> FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
>>>> RETURNING LINE:  at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 468, <GEN3> line 10.
>>>> FINISHED READING HEADERS at /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm line 312, <GEN3> line 10.
>>> 
>>> The repeated ‘RETURNING LINE: ‘… is the start of the recursive descent into oblivion.
>>> 
>>> The issue is that ResultIterator is attempting to open a Net::HTTP connection on port 443 in Webservice::InterMine::ResultIterator::connect and that fails. I’ve been seeing if I can replace this with Net::HTTPS but I’m not succeeding right away; it seems that I’m running into the Moose type checking. Being a Moose newbie, I need to see how to incorporate the netHTTPS type into the code.
>>> 
>>> Is this something that has come up before? Did you have any suggestions of other code changes or workarounds to enable the perl API over https? 
>>> 
>>> For what it’s worth, the java API is just fine. And, I’m going to put a pull request on github to correct a problem with generating the service URL for the auto generated code.
>>> 
>>> Thanks,
>>> 
>>> Joe Carlson
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.intermine.org/pipermail/dev/attachments/20160217/eba110c2/attachment-0001.html>


More information about the dev mailing list