[InterMine Dev] solr issues (again)

Joe Carlson jwcarlson at lbl.gov
Wed Jun 23 19:17:06 BST 2021


Hello,

A year or so ago I was struggling with building my solr index. It was a bit frustrating since the gradle process reports ‘BUILD SUCCESSFUL’ even though the create-search-index task itself reports 

POSTPROCESS create-search-index FAILED.
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://phytozome-next.jgi.doe.gov/ <https://phytozome-next.jgi.doe.gov/>….

The log files report this as an IOException but I’ve not been able to see more details than that. It generally happens after about an hour.

Last time I looked into this I had thought that the problem was that our solr server is sitting behind an nginx reverse proxy and nginx was limiting the body size of a post. The max size of a post body is configurable, and when I upped that to 100M, I was able to build. But, I should add, I’m indexing much fewer fields than the default. (It would just take too long to index all the collections and references.) Previously I was able to make my abbreviated index - and have a store of 11.6M documents. Now I have trouble even making an index this large; the IO exception occurs after ~7M documents.

I’m curious if anyone else has see IOExceptions when building solr. Or, from other groups, how large of an index they routinely build (and how long it takes to build it.)

Thanks,

Joe


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.intermine.org/pipermail/dev/attachments/20210623/e252de65/attachment.html>


More information about the dev mailing list