[InterMine Dev] Using perl client library with https

Josh Goodman jogoodma at indiana.edu
Thu Feb 18 06:27:35 GMT 2016


Yes, I haven't a clue about Ruby.

Josh

On Thu, Feb 18, 2016 at 1:06 AM Joe Carlson <jwcarlson at lbl.gov> wrote:

> On Feb 17, 2016, at 9:19 PM, Josh Goodman <jogoodma at indiana.edu> wrote:
>
> Hi Joe,
>
> Apologies for the incorrect syntax, I mixed up Moose vs MooseX types.  I
> am not a maintainer of the CPAN module, but I created a pull request so
> that one of the InterMine devs can take a look.
>
> https://github.com/intermine/intermine/pull/1308
>
>
> yep. this was pretty much what I did, too.
>
> Motivated by this issue, I checked the other API’s to see if they had any
> trouble with https. Java and python passed my “exhaustive" testing (1
> script each).
>
> But Ruby failed. When I tried my first script, it sent me a nastygram:
>
>
> > ruby template.rb
> /Library/Ruby/Gems/2.0.0/gems/intermine-1.04.00/lib/intermine/service.rb:173:in
> `rescue in initialize': Error fetching version at
> https://phytozome.jgi.doe.gov/phytomine/service/version: wrong status
> line: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">"
> (InterMine::ServiceError)
> from
> /Library/Ruby/Gems/2.0.0/gems/intermine-1.04.00/lib/intermine/service.rb:170:in
> `initialize'
> from template.rb:16:in `new'
> from template.rb:16:in `<main>'
>
> I know absolutely no ruby. And I’m not even certain that this is a https
> issue. But it looks like it’s getting some unexpected html when requesting
> the version. Should this be checked by some rubyist?
>
> Thanks for all your help,
>
> Joe
>
> Cheers,
> Josh
>
> On Wed, Feb 17, 2016 at 3:57 PM Joe Carlson <jwcarlson at lbl.gov> wrote:
>
>> Ah! I had added the class_type to Types.pm and I tried
>> isa=>[NetHTTP,NetHTTPS]. That didn’t work. I didnt know about the pipe
>> symbol.
>>
>> But your syntax does not work for me with the quotes in there. I have
>> more success if I write it as
>>
>> isa => NetHTTP|NetHTTPS,
>>
>> and it even work when I query flymine!
>>
>> Are you folks the maintainers of the module in CPAN? Can you push out a
>> update?
>>
>> thanks,
>>
>> Joe
>>
>> On Feb 17, 2016, at 12:12 PM, Josh Goodman <jogoodma at indiana.edu> wrote:
>>
>> Hi Joe,
>>
>> You can use type unions for this.  Given how this has been implemented
>> here, I would do the following.
>>
>> Add a new type for NetHTTPS in Webservice/InterMine/Types.pm:
>>
>> class_type NetHTTPS, { class => 'Net::HTTPS', };
>>
>> Then modify ResultIterator to import the new type and use a type union on
>> the connection attribute.
>>
>> use Webservice::InterMine::Types qw(Uri HTTPCode NetHTTP NetHTTPS
>> RowFormat JsonFormat RequestFormat RowParser);
>> ...
>> has connection => (
>>     is        => 'ro',
>>     isa       => 'NetHTTP|NetHTTPS',
>>     writer    => 'set_connection',
>>     predicate => 'has_connection',
>> );
>>
>> Josh
>>
>> On Wed, Feb 17, 2016 at 2:10 PM Joe Carlson <jwcarlson at lbl.gov> wrote:
>>
>>> For what it’s worth, I’ve verified that if I add NetHTTPS as a type in
>>> Webservice::InterMine::Types, change the ‘isa’ in connection to NetHTTPS
>>> and open the connection with Net::HTTPS, then I can use the perl API over
>>> https.
>>>
>>> But I don’t know enough about Moose to adapt the code to work with
>>> either Net::HTTP or Net::HTTPS. Line 520 in ResultIterator has a call
>>> $self->set_connection($connection) which checks that this is a specific
>>> class Net::HTTP. Is there a way to get Moose to accept alternative types in
>>> the type checking?
>>>
>>> Joe
>>>
>>>
>>> On Feb 17, 2016, at 10:15 AM, Joe Carlson <jwcarlson at lbl.gov> wrote:
>>>
>>> New problem.
>>>
>>> We have a mandate to use https exclusively for our web services. One of
>>> our users has seen that the perl client libraries do not work at our site.
>>> We have confirmed that this is a problem. There are some warning messages
>>> about deep recursion on subroutine
>>> "Webservice::InterMine::ResultIterator::set_headers” and the script never
>>> returns results.
>>>
>>> I discovered the trick of setting the DEBUG environment variable to get
>>> some verbosity in the output. The first few fetches seem to work fine.
>>> There are some calls to fetch the service version and model. That goes
>>> well.Here is the output:
>>>
>>> FETCHING https://phytozome.jgi.doe.gov/phytomine/service/version
>>> 1455730008.88403 at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line
>>> 830.
>>> FINISHED FETCHING
>>> https://phytozome.jgi.doe.gov/phytomine/service/version
>>> 1455730009.32112 at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line
>>> 839.
>>> FETCHING
>>> https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml
>>> 1455730009.32364 at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line
>>> 830.
>>> FINISHED FETCHING
>>> https://phytozome.jgi.doe.gov/phytomine/service/model?format=xml
>>> 1455730011.81447 at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/Service.pm line
>>> 839.
>>> string=><model name="genomic" package="org.intermine.model.bio">
>>> <class name="DatabaseReference" is-interface="true">
>>>         <attribute name="identifier" type="java.lang.String"/>
>>>         <reference name="subject" referenced-type="Publication"
>>> reverse-reference="crossReferences"/>
>>>         <reference name="source" referenced-type="DataSource”/>
>>>
>>> ….
>>>
>>> This went well and it appears to use the https connection retrieve the
>>> contents.
>>>
>>>
>>> CONNECTING TO phytozome.jgi.doe.gov:443 at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 493.
>>> UNENCODED-PARAM: op1==
>>> UNENCODED-PARAM: constraint1=Gene.primaryIdentifier
>>> UNENCODED-PARAM: format=json
>>> UNENCODED-PARAM: name=GO_terms_for_genes
>>> UNENCODED-PARAM: value1=Potri.014G066100
>>> UNENCODED-PARAM: code1=A
>>> SENDING REQUESTPOST /phytomine/service/template/results HTTP/1.1
>>> Connection: close
>>> Host: phytozome.jgi.doe.gov:443
>>> Content-Type: application/x-www-form-urlencoded
>>> User-Agent: Webservice::InterMine-1.0405/Perl client library
>>> Content-Length: 110
>>>
>>>
>>> op1=%3D&constraint1=Gene.primaryIdentifier&format=json&name=GO_terms_for_genes&value1=Potri.014G066100&code1=A
>>> SETTING HEADERS at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 26.
>>> RETURNING LINE: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 1.
>>> RETURNING LINE: <html><head> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 2.
>>> RETURNING LINE: <title>400 Bad Request</title> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 3.
>>> RETURNING LINE: </head><body> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 4.
>>> RETURNING LINE: <h1>Bad Request</h1> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 5.
>>> RETURNING LINE: <p>Your browser sent a request that this server could
>>> not understand.<br /> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 6.
>>> RETURNING LINE: Reason: You're speaking plain HTTP to an SSL-enabled
>>> server port.<br /> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 7.
>>> HEADER Reason = You're speaking plain HTTP to an SSL-enabled server
>>> port.<br /> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 308, <GEN3> line 7.
>>> RETURNING LINE: Instead use the HTTPS scheme to access this URL,
>>> please.<br /> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 8.
>>> RETURNING LINE: <blockquote>Hint: <a href="https://gnweb01.nersc.gov/
>>> "><b>https://gnweb01.nersc.gov/</b></a></blockquote></p> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 9.
>>> HEADER <blockquote>Hint = <a href="https://gnweb01.nersc.gov/"><b>
>>> https://gnweb01.nersc.gov/</b></a></blockquote></p> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 308, <GEN3> line 9.
>>> RETURNING LINE: </body></html> at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 10.
>>> RETURNING LINE:  at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 10.
>>> FINISHED READING HEADERS at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 312, <GEN3> line 10.
>>> RETURNING LINE:  at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 10.
>>> FINISHED READING HEADERS at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 312, <GEN3> line 10.
>>> RETURNING LINE:  at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 10.
>>> FINISHED READING HEADERS at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 312, <GEN3> line 10.
>>> RETURNING LINE:  at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 468, <GEN3> line 10.
>>> FINISHED READING HEADERS at
>>> /opt/local/lib/perl5/site_perl/5.22/Webservice/InterMine/ResultIterator.pm
>>> line 312, <GEN3> line 10.
>>>
>>>
>>> The repeated ‘RETURNING LINE: ‘… is the start of the recursive descent
>>> into oblivion.
>>>
>>> The issue is that ResultIterator is attempting to open a Net::HTTP
>>> connection on port 443 in Webservice::InterMine::ResultIterator::connect
>>> and that fails. I’ve been seeing if I can replace this with Net::HTTPS but
>>> I’m not succeeding right away; it seems that I’m running into the Moose
>>> type checking. Being a Moose newbie, I need to see how to incorporate the
>>> netHTTPS type into the code.
>>>
>>> Is this something that has come up before? Did you have any suggestions
>>> of other code changes or workarounds to enable the perl API over https?
>>>
>>> For what it’s worth, the java API is just fine. And, I’m going to put a
>>> pull request on github to correct a problem with generating the service URL
>>> for the auto generated code.
>>>
>>> Thanks,
>>>
>>> Joe Carlson
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.intermine.org/pipermail/dev/attachments/20160218/8f4ae682/attachment-0001.html>


More information about the dev mailing list