[InterMine Dev] solr issues (again)

Joe Carlson jwcarlson at lbl.gov
Wed Jun 23 19:17:06 BST 2021


A year or so ago I was struggling with building my solr index. It was a bit frustrating since the gradle process reports ‘BUILD SUCCESSFUL’ even though the create-search-index task itself reports 

POSTPROCESS create-search-index FAILED.
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://phytozome-next.jgi.doe.gov/ <https://phytozome-next.jgi.doe.gov/>….

The log files report this as an IOException but I’ve not been able to see more details than that. It generally happens after about an hour.

Last time I looked into this I had thought that the problem was that our solr server is sitting behind an nginx reverse proxy and nginx was limiting the body size of a post. The max size of a post body is configurable, and when I upped that to 100M, I was able to build. But, I should add, I’m indexing much fewer fields than the default. (It would just take too long to index all the collections and references.) Previously I was able to make my abbreviated index - and have a store of 11.6M documents. Now I have trouble even making an index this large; the IO exception occurs after ~7M documents.

I’m curious if anyone else has see IOExceptions when building solr. Or, from other groups, how large of an index they routinely build (and how long it takes to build it.)



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.intermine.org/pipermail/dev/attachments/20210623/e252de65/attachment.html>

More information about the dev mailing list