Discussion:
[Exist-open] Switching to read-only mode!!!
Nick Sincaglia
2017-06-07 20:44:10 UTC
Permalink
I am running eXist-db 3.0 RC1 with a Linux AWS OS on a production server. Recently, I have been having to needing to restart it manually once a day or it will eventually switch to read-only on me. I have looked through wrapper.log and exist.log for hints of why this might be happening. I have chased down a few theories and made some adjustments to my code but so far, the behavior has not changed. I was hoping that if I posted the error messages I am seeing in exist.log, someone might be able to give me a hint as to why this is happening. I can see that eXist-db is complaining that there are too many files open in jetty/tmp. Is there anything that might be causing this? Does anyone have any suggestions on how I might identify the root cause of this issue? I am hoping someone with more knowledge about eXist-db might be able to suggest a direction in which I should look for answers. I don’t have any clue at the moment of what is causing this issue.

Nick

2017-06-07 18:32:01,747 [DefaultQuartzScheduler_Worker-1] INFO (NativeBroker.java [sync]:3594) - Memory: 4,700,160K total; 5,592,576K max; 1,221,496K free
2017-06-07 18:32:01,747 [DefaultQuartzScheduler_Worker-1] INFO (BTree.java [printStatistics]:2540) - dom.dbx INDEX Buffers occupation : 100% (64 out of 64) Cache efficiency : 100%
2017-06-07 18:32:01,747 [DefaultQuartzScheduler_Worker-1] INFO (DOMFile.java [printStatistics]:1358) - dom.dbx DATA Buffers occupation : 100% (256 out of 256) Cache efficiency : 26%
2017-06-07 18:32:01,748 [DefaultQuartzScheduler_Worker-1] INFO (BTree.java [printStatistics]:2540) - collections.dbx INDEX Buffers occupation : 72% (104 out of 144) Cache efficiency : 99%
2017-06-07 18:32:01,748 [DefaultQuartzScheduler_Worker-1] INFO (BFile.java [printStatistics]:433) - collections.dbx DATA Buffers occupation : 88% (266 out of 303) Cache efficiency : 98%
2017-06-07 18:32:01,748 [DefaultQuartzScheduler_Worker-1] INFO (BTree.java [printStatistics]:2540) - values.dbx INDEX Buffers occupation : 2% (1 out of 64) Cache efficiency : 100%
2017-06-07 18:32:01,748 [DefaultQuartzScheduler_Worker-1] INFO (BFile.java [printStatistics]:433) - values.dbx DATA Buffers occupation : 0% (0 out of 64) Cache efficiency : N/A
2017-06-07 18:32:24,247 [DefaultQuartzScheduler_Worker-2] INFO (BrokerPool.java [setReadOnly]:1333) - Switching to read-only mode!!!
2017-06-07 18:32:25,349 [DefaultQuartzScheduler_Worker-3] ERROR (FilterInputStreamCacheFactory.java [instantiate]:81) - null
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor118.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_65]
at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_65]
at org.exist.util.io.FilterInputStreamCacheFactory.instantiate(FilterInputStreamCacheFactory.java:72) [exist.jar:?]
at org.exist.util.io.FilterInputStreamCacheFactory.getCacheInstance(FilterInputStreamCacheFactory.java:60) [exist.jar:?]
at org.exist.xquery.value.BinaryValueFromInputStream.<init>(BinaryValueFromInputStream.java:33) [exist.jar:?]
at org.exist.xquery.value.BinaryValueFromInputStream.getInstance(BinaryValueFromInputStream.java:51) [exist.jar:?]
at org.expath.httpclient.model.exist.EXistResult.add(EXistResult.java:97) [exist-expath-20130805.jar:?]
at org.expath.httpclient.impl.BinaryResponseBody.<init>(BinaryResponseBody.java:45) [http-client-java-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at org.expath.httpclient.impl.BodyFactory.makeResponsePart(BodyFactory.java:105) [http-client-java-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at org.expath.httpclient.impl.BodyFactory.makeResponseBody(BodyFactory.java:85) [http-client-java-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at org.expath.httpclient.impl.HttpRequestImpl.send(HttpRequestImpl.java:63) [http-client-java-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at org.expath.exist.SendRequestFunction.sendOnce(SendRequestFunction.java:178) [exist-expath-20130805.jar:?]
at org.expath.exist.SendRequestFunction.sendRequest(SendRequestFunction.java:151) [exist-expath-20130805.jar:?]
at org.expath.exist.SendRequestFunction.eval(SendRequestFunction.java:133) [exist-expath-20130805.jar:?]
at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70) [exist.jar:?]
at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:56) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.ForExpr.processItem(ForExpr.java:238) [exist.jar:?]
at org.exist.xquery.ForExpr.eval(ForExpr.java:178) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:74) [exist.jar:?]
at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:74) [exist.jar:?]
at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:280) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:209) [exist.jar:?]
at org.exist.xquery.functions.util.Eval.execute(Eval.java:442) [exist.jar:?]
at org.exist.xquery.functions.util.Eval.doEval(Eval.java:396) [exist.jar:?]
at org.exist.xquery.functions.util.Eval.eval(Eval.java:225) [exist.jar:?]
at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70) [exist.jar:?]
at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:56) [exist.jar:?]
at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:74) [exist.jar:?]
at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
at org.exist.xquery.TryCatchExpression.eval(TryCatchExpression.java:127) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:280) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:209) [exist.jar:?]
at org.exist.xquery.functions.util.Eval.execute(Eval.java:442) [exist.jar:?]
at org.exist.xquery.functions.util.Eval.doEval(Eval.java:396) [exist.jar:?]
at org.exist.xquery.functions.util.Eval.eval(Eval.java:225) [exist.jar:?]
at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70) [exist.jar:?]
at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:56) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
at org.exist.xquery.FunctionCall$DeferredFunctionCallImpl.execute(FunctionCall.java:427) [exist.jar:?]
at org.exist.xquery.DeferredFunctionCall.realize(DeferredFunctionCall.java:57) [exist.jar:?]
at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:198) [exist.jar:?]
at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:76) [exist.jar:?]
at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:280) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:200) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:195) [exist.jar:?]
at org.exist.scheduler.UserXQueryJob.execute(UserXQueryJob.java:217) [exist.jar:?]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.1.jar:?]
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.1.jar:?]
Caused by: java.nio.file.FileSystemException: ../../../tools/jetty/tmp/_mmtfm_3c7e7f9b-9900-4444-9f03-a9d850829b0c1632133065744495093/mmtf_14968603453487918292215225476280.tmp: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:1.8.0_65]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_65]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_65]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:1.8.0_65]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_65]
at java.nio.file.Files.createFile(Files.java:632) ~[?:1.8.0_65]
at java.nio.file.TempFileHelper.create(TempFileHelper.java:138) ~[?:1.8.0_65]
at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161) ~[?:1.8.0_65]
at java.nio.file.Files.createTempFile(Files.java:852) ~[?:1.8.0_65]
at org.exist.util.io.TemporaryFileManager.getTemporaryFile(TemporaryFileManager.java:95) ~[exist.jar:?]
at org.exist.util.io.FileFilterInputStreamCache.<init>(FileFilterInputStreamCache.java:61) ~[exist.jar:?]
at org.exist.util.io.FileFilterInputStreamCache.<init>(FileFilterInputStreamCache.java:55) ~[exist.jar:?]
... 145 more
2017-06-07 18:32:25,494 [DefaultQuartzScheduler_Worker-3] ERROR (XMLDBCreateCollection.java [evalWithCollection]:87) - Unable to create new collection 20170607
org.xmldb.api.base.XMLDBException: database is read-only
at org.exist.xmldb.function.LocalXmldbFunction.apply(LocalXmldbFunction.java:48) ~[exist.jar:?]
at org.exist.xmldb.AbstractLocal.withDb(AbstractLocal.java:193) ~[exist.jar:?]
at org.exist.xmldb.LocalCollectionManagementService.createCollection(LocalCollectionManagementService.java:81) ~[exist.jar:?]
at org.exist.xmldb.LocalCollectionManagementService.createCollection(LocalCollectionManagementService.java:71) ~[exist.jar:?]
at org.exist.xmldb.LocalCollectionManagementService.createCollection(LocalCollectionManagementService.java:60) ~[exist.jar:?]
at org.exist.xquery.functions.xmldb.XMLDBAbstractCollectionManipulator.createCollection(XMLDBAbstractCollectionManipulator.java:199) ~[exist.jar:?]
at org.exist.xquery.functions.xmldb.XMLDBAbstractCollectionManipulator.createCollectionPath(XMLDBAbstractCollectionManipulator.java:209) ~[exist.jar:?]
at org.exist.xquery.functions.xmldb.XMLDBCreateCollection.evalWithCollection(XMLDBCreateCollection.java:79) [exist.jar:?]
at org.exist.xquery.functions.xmldb.XMLDBAbstractCollectionManipulator.eval(XMLDBAbstractCollectionManipulator.java:174) [exist.jar:?]
at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70) [exist.jar:?]
at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:56) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:102) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:74) [exist.jar:?]
at org.exist.xquery.Atomize.eval(Atomize.java:67) [exist.jar:?]
at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
at org.exist.xquery.TryCatchExpression.eval(TryCatchExpression.java:127) [exist.jar:?]
at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:65) [exist.jar:?]
at org.exist.xquery.ForExpr.processItem(ForExpr.java:238) [exist.jar:?]
at org.exist.xquery.ForExpr.eval(ForExpr.java:178) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:280) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:209) [exist.jar:?]
at org.exist.xquery.functions.util.Eval.execute(Eval.java:442) [exist.jar:?]
at org.exist.xquery.functions.util.Eval.doEval(Eval.java:396) [exist.jar:?]
at org.exist.xquery.functions.util.Eval.eval(Eval.java:225) [exist.jar:?]
at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70) [exist.jar:?]
at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:56) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
at org.exist.xquery.FunctionCall$DeferredFunctionCallImpl.execute(FunctionCall.java:427) [exist.jar:?]
at org.exist.xquery.DeferredFunctionCall.realize(DeferredFunctionCall.java:57) [exist.jar:?]
at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:198) [exist.jar:?]
at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:199) [exist.jar:?]
at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:199) [exist.jar:?]
at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:199) [exist.jar:?]
at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:199) [exist.jar:?]
at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:76) [exist.jar:?]
at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:280) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:200) [exist.jar:?]
at org.exist.xquery.XQuery.execute(XQuery.java:195) [exist.jar:?]
at org.exist.scheduler.UserXQueryJob.execute(UserXQueryJob.java:217) [exist.jar:?]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.1.jar:?]
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.1.jar:?]
Caused by: org.exist.security.PermissionDeniedException: database is read-only
at org.exist.storage.NativeBroker.getOrCreateCollectionExplicit(NativeBroker.java:738) ~[exist.jar:?]
at org.exist.storage.NativeBroker.getOrCreateCollection(NativeBroker.java:664) ~[exist.jar:?]
at org.exist.xmldb.LocalCollectionManagementService.lambda$createCollection$234(LocalCollectionManagementService.java:83) ~[exist.jar:?]
at org.exist.xmldb.function.LocalXmldbFunction.apply(LocalXmldbFunction.java:46) ~[exist.jar:?]
... 88 more
2017-06-07 18:34:04,247 [DefaultQuartzScheduler_Worker-4] FATAL (Journal.java [clearBackupFiles]:381) - Could not clear journal backup files
java.nio.file.FileSystemException: /mnt/data/exist-3.0-RC/tools/wrapper/bin/../../../webapp/WEB-INF/data/fs.journal: Too many open files
Joe Wicentowski
2017-06-08 01:24:55 UTC
Permalink
Hi Nick,

You might find the troubleshooting techniques described in this thread
from exist-open from Oct. 2015 useful:

http://markmail.org/message/cuwcfbnh6hg5vvwj

See also this article (mentioned in the same thread):

https://underyx.me/2015/05/18/raising-the-maximum-number-of-file-descriptors.html

One cause of exceeding the limit on open files can be failing to close
HTTP connections made via eXist's native httpclient or EXPath HTTP
Client modules. Can you check your HTTP requests and make sure you're
including the "Connection: Close" header?

Joe
I am running eXist-db 3.0 RC1 with a Linux AWS OS on a production server. Recently, I have been having to needing to restart it manually once a day or it will eventually switch to read-only on me... I can see that eXist-db is complaining that there are too many files open in jetty/tmp.
...
2017-06-07 18:34:04,247 [DefaultQuartzScheduler_Worker-4] FATAL (Journal.java [clearBackupFiles]:381) - Could not clear journal backup files
java.nio.file.FileSystemException: /mnt/data/exist-3.0-RC/tools/wrapper/bin/../../../webapp/WEB-INF/data/fs.journal: Too many open files
nsincaglia
2017-06-09 02:39:46 UTC
Permalink
Joe,

Thanks for the suggestion. We are using the EXPath HTTP Client Module but we
are not including the Connection:close header. That is something I will
implement and I will report back if it corrects the issue.

Nick




--
View this message in context: http://exist.2174344.n4.nabble.com/Switching-to-read-only-mode-tp4672288p4672298.html
Sent from the exist-open mailing list archive at Nabble.com.
nsincaglia
2017-06-20 15:46:22 UTC
Permalink
I just wanted to report back on this issue. We added the Connection: Close
header to all of our Expath Http-Client requests and after monitoring it for
several days now, it appears to have corrected the issue, which we are very
happy about. Thanks for the suggestion!!!

What I find puzzling is why this is not documented anywhere. I looked
through the EXPath HTTP-Client Module documentation and I don't see any
references at all to the Connection:Close header.

I am also trying to understand why the default is to keep the connection
open. How would one even use a previous http request in XQuery? There is no
handle or reference one can use, I don't think. I am wondering why the
default is to not close the connection?

Does anyone have any insights on this?

Nick





--
View this message in context: http://exist.2174344.n4.nabble.com/Switching-to-read-only-mode-tp4672288p4672338.html
Sent from the exist-open mailing list archive at Nabble.com.
Adam Retter
2017-06-22 13:30:27 UTC
Permalink
Would anyone be interested in volunteering to add a PR to the documentation
repo to mention this?
Post by nsincaglia
I just wanted to report back on this issue. We added the Connection: Close
header to all of our Expath Http-Client requests and after monitoring it for
several days now, it appears to have corrected the issue, which we are very
happy about. Thanks for the suggestion!!!
What I find puzzling is why this is not documented anywhere. I looked
through the EXPath HTTP-Client Module documentation and I don't see any
references at all to the Connection:Close header.
I am also trying to understand why the default is to keep the connection
open. How would one even use a previous http request in XQuery? There is no
handle or reference one can use, I don't think. I am wondering why the
default is to not close the connection?
Does anyone have any insights on this?
Nick
--
View this message in context: http://exist.2174344.n4.
nabble.com/Switching-to-read-only-mode-tp4672288p4672338.html
Sent from the exist-open mailing list archive at Nabble.com.
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
https://lists.sourceforge.net/lists/listinfo/exist-open
--
Adam Retter

eXist Developer
{ United Kingdom }
***@exist-db.org
irc://irc.freenode.net/existdb
Continue reading on narkive:
Loading...