[InterMine Dev] solr issues (again)
jwcarlson at lbl.gov
Wed Jun 23 19:17:06 BST 2021
A year or so ago I was struggling with building my solr index. It was a bit frustrating since the gradle process reports ‘BUILD SUCCESSFUL’ even though the create-search-index task itself reports
POSTPROCESS create-search-index FAILED.
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://phytozome-next.jgi.doe.gov/ <https://phytozome-next.jgi.doe.gov/>….
The log files report this as an IOException but I’ve not been able to see more details than that. It generally happens after about an hour.
Last time I looked into this I had thought that the problem was that our solr server is sitting behind an nginx reverse proxy and nginx was limiting the body size of a post. The max size of a post body is configurable, and when I upped that to 100M, I was able to build. But, I should add, I’m indexing much fewer fields than the default. (It would just take too long to index all the collections and references.) Previously I was able to make my abbreviated index - and have a store of 11.6M documents. Now I have trouble even making an index this large; the IO exception occurs after ~7M documents.
I’m curious if anyone else has see IOExceptions when building solr. Or, from other groups, how large of an index they routinely build (and how long it takes to build it.)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the dev