Hi, Not sure if there is a better forum for this. We're running collectionspace 0.7 on Redhat Enterprise Linix 5.5. Last night the default (nuxeo?) server shutdown for no apparent reason. 2010-06-21 02:00:00,016 INFO [org.nuxeo.ecm.platform.scheduler.core.EventJob] Sending scheduled event id=MailReceivedEvent, category=default 2010-06-21 02:30:00,019 INFO [org.nuxeo.ecm.platform.scheduler.core.EventJob] Sending scheduled event id=MailReceivedEvent, category=default 2010-06-21 02:35:35,664 INFO [org.jboss.system.server.Server] Runtime shutdown hook called, forceHalt: true 2010-06-21 02:35:35,665 INFO [org.jboss.system.server.Server] JBoss SHUTDOWN: Undeploying all packages If it helps, here are utility (java, ant, maven) versions. [root@theoden log]# java -version java version "1.6.0_20" Java(TM) SE Runtime Environment (build 1.6.0_20-b02) Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing) [root@theoden log]# ant -version Apache Ant version 1.8.1 compiled on April 30 2010 [root@theoden log]# mvn -version Apache Maven 2.2.1 (r801777; 2009-08-06 12:16:01-0700) Java version: 1.6.0_20 Java home: /usr/local/jdk1.6.0_20/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux" version: "2.6.18-194.3.1.el5" arch: "i386" Family: "unix" If you want other logs, let me know. Thanks, Ron

Hi Ron, On Mon, Jun 21, 2010 at 2:58 PM, Stanonik, Ronald <rstanonik@ucsd.edu> wrote: > Not sure if there is a better forum for this. Not sure either - am copying the CollectionSpace 'tech' list in case some members of that list might not be subscribed to the current 'talk' list. Perhaps for now, we might continue this conversation on both lists. > Last night the default (nuxeo?) server shutdown for no apparent reason. Weird. One data point that may help to some extent: I can't recall personally experiencing a spontaneous shutdown of either of the two JBoss server domains (nuxeo [currently in 'servers/default'] or cspace). This is based on running various tagged and snapshot versions of the services layer over a period of about nine months, under Fedora Linux 10. There are a couple of threads in the JBoss forums about unexpected shutdowns. The most useful post appears to be from Adrian Brock (https://jira.jboss.org/browse/JBAS-3237), who is a lead on several JBoss projects (http://community.jboss.org/people/adrian@jboss.org): "Something is invoking System.exit(), it isn't JBoss unless it is via an (unsecured?) jmx-console. Or the OS is sending a term [kill - Aron] signal." An earlier comment suggests: "enable TRACE for org.jboss.system.server.Server to hopefully get some more info, and try find who maybe sending kill signals to the java process, or remote client shutdown/exit calls." It appears you might be able to add a log4j configuration file to the nuxeo (default) server, as $JBOSS_HOME/server/default/conf/log4j.xml, that would enable TRACE-level logging for the org.jboss.system.server.Server class. Also, you might make sure that the JBoss consoles are either disabled or secured, so that someone/something isn't shutting down servers via those consoles. To the best of my knowledge, they should be disabled by default, in the two-server .gz.tar distribution available via FTP from source.collectionspace.org, through removal of the consoles' WAR files from the deploy directory, as described in points 1) and 2) in this comment: http://issues.collectionspace.org/browse/CSPACE-344?focusedCommentId=12069&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12069 Aron -- > > We're running collectionspace 0.7 on Redhat Enterprise Linix 5.5. > > Last night the default (nuxeo?) server shutdown for no apparent reason. > > > 2010-06-21 02:00:00,016 INFO  [org.nuxeo.ecm.platform.scheduler.core.EventJob] Sending scheduled event id=MailReceivedEvent, category=default > 2010-06-21 02:30:00,019 INFO  [org.nuxeo.ecm.platform.scheduler.core.EventJob] Sending scheduled event id=MailReceivedEvent, category=default > 2010-06-21 02:35:35,664 INFO  [org.jboss.system.server.Server] Runtime shutdown hook called, forceHalt: true > 2010-06-21 02:35:35,665 INFO  [org.jboss.system.server.Server] JBoss SHUTDOWN: Undeploying all packages > > > If it helps, here are utility (java, ant, maven) versions. > > [root@theoden log]# java -version > java version "1.6.0_20" > Java(TM) SE Runtime Environment (build 1.6.0_20-b02) > Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing) > [root@theoden log]# ant -version > Apache Ant version 1.8.1 compiled on April 30 2010 > [root@theoden log]# mvn -version > Apache Maven 2.2.1 (r801777; 2009-08-06 12:16:01-0700) > Java version: 1.6.0_20 > Java home: /usr/local/jdk1.6.0_20/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux" version: "2.6.18-194.3.1.el5" arch: "i386" Family: "unix" > > If you want other logs, let me know. > > Thanks, > > Ron > > > _______________________________________________ > Talk mailing list > Talk@lists.collectionspace.org > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org >

Hi Ron, To help investigate the possibility that the shutdown of the Nuxeo server might be related to the action of code running in the second (CSpace) server, which acts as a client of the Nuxeo server, Patrick Schmitz suggested that you can also send us the relevant log files from the CSpace server, or perhaps excerpts from those files spanning the time that the unexpected server shutdown occurred. You can post excerpts here to the lists, attach them, or just send them to me privately, if you prefer. You can find these in: $JBOSS_HOME/server/cspace/log The relevant files are: - server.log - collectionspace.log These logs are rotated daily, so it's possible that the relevant time period might be represented by date-stamped log files, such as server.log.2010-06-21 and collectionspace.log.2010-06-21. Aron On Mon, Jun 21, 2010 at 6:30 PM, Aron Roberts <aron@socrates.berkeley.edu> wrote: > On Mon, Jun 21, 2010 at 2:58 PM, Stanonik, Ronald <rstanonik@ucsd.edu> wrote: >> Not sure if there is a better forum for this. > >  Not sure either - am copying the CollectionSpace 'tech' list in case > some members of that list might not be subscribed to the current > 'talk' list.  Perhaps for now, we might continue this conversation on > both lists. > >> Last night the default (nuxeo?) server shutdown for no apparent reason. > >  Weird.  One data point that may help to some extent: I can't recall > personally experiencing a spontaneous shutdown of either of the two > JBoss server domains (nuxeo [currently in 'servers/default'] or > cspace).  This is based on running various tagged and snapshot > versions of the services layer over a period of about nine months, > under Fedora Linux 10. > >  There are a couple of threads in the JBoss forums about unexpected > shutdowns.  The most useful post appears to be from Adrian Brock > (https://jira.jboss.org/browse/JBAS-3237), who is a lead on several > JBoss projects (http://community.jboss.org/people/adrian@jboss.org): > > "Something is invoking System.exit(), it isn't JBoss unless it is via > an (unsecured?) jmx-console. > Or the OS is sending a term [kill - Aron] signal." > >  An earlier comment suggests: > > "enable TRACE for org.jboss.system.server.Server to hopefully get some > more info, and try find who maybe sending kill signals to the java > process, or remote client shutdown/exit calls." > >  It appears you might be able to add a log4j configuration file to > the nuxeo (default) server, as > $JBOSS_HOME/server/default/conf/log4j.xml, that would enable > TRACE-level logging for the org.jboss.system.server.Server class. > >  Also, you might make sure that the JBoss consoles are either > disabled or secured, so that someone/something isn't shutting down > servers via those consoles.  To the best of my knowledge, they should > be disabled by default, in the two-server .gz.tar distribution > available via FTP from source.collectionspace.org, through removal of > the consoles' WAR files from the deploy directory, as described in > points 1) and 2) in this comment: > > http://issues.collectionspace.org/browse/CSPACE-344?focusedCommentId=12069&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12069 > > Aron > > -- > >> >> We're running collectionspace 0.7 on Redhat Enterprise Linix 5.5. >> >> Last night the default (nuxeo?) server shutdown for no apparent reason. >> >> >> 2010-06-21 02:00:00,016 INFO  [org.nuxeo.ecm.platform.scheduler.core.EventJob] Sending scheduled event id=MailReceivedEvent, category=default >> 2010-06-21 02:30:00,019 INFO  [org.nuxeo.ecm.platform.scheduler.core.EventJob] Sending scheduled event id=MailReceivedEvent, category=default >> 2010-06-21 02:35:35,664 INFO  [org.jboss.system.server.Server] Runtime shutdown hook called, forceHalt: true >> 2010-06-21 02:35:35,665 INFO  [org.jboss.system.server.Server] JBoss SHUTDOWN: Undeploying all packages >> >> >> If it helps, here are utility (java, ant, maven) versions. >> >> [root@theoden log]# java -version >> java version "1.6.0_20" >> Java(TM) SE Runtime Environment (build 1.6.0_20-b02) >> Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing) >> [root@theoden log]# ant -version >> Apache Ant version 1.8.1 compiled on April 30 2010 >> [root@theoden log]# mvn -version >> Apache Maven 2.2.1 (r801777; 2009-08-06 12:16:01-0700) >> Java version: 1.6.0_20 >> Java home: /usr/local/jdk1.6.0_20/jre >> Default locale: en_US, platform encoding: UTF-8 >> OS name: "linux" version: "2.6.18-194.3.1.el5" arch: "i386" Family: "unix" >> >> If you want other logs, let me know. >> >> Thanks, >> >> Ron >> >> >> _______________________________________________ >> Talk mailing list >> Talk@lists.collectionspace.org >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org >> >

Hi Ron, I have experienced the same problem. Every night, a couple of hours after I left the office the servers would shut down with the same shutdown message you have seen. In my case the problem was that I'd been running the jboss processes as background jobs and when my remote ssh session to the server was logged off (after timeout) a stop signal was being sent to those processes. To prevent this I now start both jboss processes with the 'nohup' command. Something like this... nohup run.sh -b 0.0.0.0 &> nuxeo.log & nohup run.sh -b 0.0.0.0 -c cspace &> cspace.log & /Chris Developer, Corpus Project Statens Museum for Kunst (National Gallery of Denmark) -----Oprindelig meddelelse----- Fra: talk-bounces@lists.collectionspace.org [mailto:talk-bounces@lists.collectionspace.org] På vegne af Stanonik, Ronald Sendt: 21. juni 2010 23:59 Til: talk@lists.collectionspace.org Emne: [Talk] Runtime shutdown hook called. Hi, Not sure if there is a better forum for this. We're running collectionspace 0.7 on Redhat Enterprise Linix 5.5. Last night the default (nuxeo?) server shutdown for no apparent reason. 2010-06-21 02:00:00,016 INFO [org.nuxeo.ecm.platform.scheduler.core.EventJob] Sending scheduled event id=MailReceivedEvent, category=default 2010-06-21 02:30:00,019 INFO [org.nuxeo.ecm.platform.scheduler.core.EventJob] Sending scheduled event id=MailReceivedEvent, category=default 2010-06-21 02:35:35,664 INFO [org.jboss.system.server.Server] Runtime shutdown hook called, forceHalt: true 2010-06-21 02:35:35,665 INFO [org.jboss.system.server.Server] JBoss SHUTDOWN: Undeploying all packages If it helps, here are utility (java, ant, maven) versions. [root@theoden log]# java -version java version "1.6.0_20" Java(TM) SE Runtime Environment (build 1.6.0_20-b02) Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing) [root@theoden log]# ant -version Apache Ant version 1.8.1 compiled on April 30 2010 [root@theoden log]# mvn -version Apache Maven 2.2.1 (r801777; 2009-08-06 12:16:01-0700) Java version: 1.6.0_20 Java home: /usr/local/jdk1.6.0_20/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux" version: "2.6.18-194.3.1.el5" arch: "i386" Family: "unix" If you want other logs, let me know. Thanks, Ron _______________________________________________ Talk mailing list Talk@lists.collectionspace.org http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org

From: Christopher Pott [Christopher.Pott@smk.dk] I have experienced the same problem. Every night, a couple of hours after I left the office the servers would shut down with the same shutdown message you have seen. In my case the problem was that I'd been running the jboss processes as background jobs and when my remote ssh session to the server was logged off (after timeout) a stop signal was being sent to those processes. To prevent this I now start both jboss processes with the 'nohup' command. Something like this... nohup run.sh -b 0.0.0.0 &> nuxeo.log & nohup run.sh -b 0.0.0.0 -c cspace &> cspace.log & Bingo! I started the jboss from my work desktop machine, which crashed around 2am. Apparently taking jboss with it. I'll start them with nohup in the future Thanks, Ron

From: Christopher Pott [Christopher.Pott@smk.dk] > I have experienced the same problem. Every night, a couple of hours > after I left the office the servers would shut down with the same > shutdown message you have seen. > > In my case the problem was that I'd been running the jboss processes > as background jobs and when my remote ssh session to the server > was logged off (after timeout) a stop signal was being sent to those processes. On Tue, Jun 22, 2010 at 5:43 AM, Stanonik, Ronald <rstanonik@ucsd.edu> wrote: > I started the jboss from my work desktop machine, which crashed around 2am. > Apparently taking jboss with it. I'll start them with nohup in the future Thanks much for your reports, Chris and Ron! This issue is now documented here: "Nuxeo or CSpace server spontaneously quits / shuts down" http://issues.collectionspace.org/browse/CSPACE-2193 Aron