Some thoughts: regardless of what you see visually in the UI or its
behavior, the 'ground truth' of what data is available to a CollectionSpace
system can be identified by looking at database tables, first and foremost.
And in nearly all cases, you should see the same data by querying the
Services REST APIs.

0. Does the 'missing' data - user accounts, storage locations, dynasties
   (concept authority terms?) appear as you might expect when viewing database
   data?

This usually involves connecting to your PostgreSQL database server via a
client like 'psql' (or the like), changing to the relevant
{tenantname_default} database, and viewing relevant tables; e.g. something
like:

$ sudo su - postgres
$ psql

postgres=#  -- List databases
postgres=# \l

postgres=#  -- Change to the database for the 'fcart' tenant
postgres=# \c fcart_default

postgres=#  -- View storage location records
fcart_default=# select * from locations_common;

postgres=#  -- Change to the 'cspace' database (AuthN/AuthZ data)
fcart_default=# \c cspace_default

fcart_default=#  -- View user accounts
fcart_default=# select * from accounts_common;

If so, then there are three questions that come to mind:

1. A collection of similar questions: Is the currently running server
   pointed at that data? Was a different server instance started? Has the URL
   you're using been redirected (such as via Apache HTTP Server or Tomcat) to
   a different server instance? Is the 'correct' server pointed at the
   database server instance you're expecting, or a different one (host/port,
   or?)

Regarding all or most of these questions, using the Services REST APIs (
https://wiki.collectionspace.org/display/DOC/Services+Layer+REST+API) can
usually help you spot any anomalies. Calls to
http://host:port/cspace-services/some_service_name_here;
e.g. http://host:port/cspace-services/locationauthorities/{csid}/items can
usually verify that you're seeing what you expect to see.

If database data looks reasonably like what you expect, and the Services
REST API calls also return that data, then:

2. Is the CollectionSpace Application layer pointed at the correct Services
   layer?

It's rare this would be changed by server upgrades or other routine
maintenance work, but you can nonetheless look at the current configuration
in the 'persistence/service' section of
$CSPACE_JEESERVER_HOME/lib/defaults/settings.xml

3. Is there some disconnect between the data displayed in your browser and
   what's being sent to it from the Application layer?

CollectionSpace caches a fair number of artifacts to improve performance (
https://wiki.collectionspace.org/display/DOC/Configuring+Caching+in+CollectionSpace).
On at least a few occasions, most notably after server software upgrades,
stale cache data can be problematic.

As a time-saving expedient, one possibility might be to start with 3),
then, if that doesn't help, work your way up from 0) through 2).

You can start with 3) by once again clearing all browser caches - including
both the browser's own file cache and its offline application cache, by
completely following the instructions at
https://wiki.collectionspace.org/display/DOC/Clearing+cached+content+from+browsers

Aron

On Wed, Jun 15, 2016 at 4:39 PM, James Keeline james@keeline.com wrote:

issue.

Running the Ant 'create_db' target with the '-Drecreate_db=true' flag drops
and re-creates databases, which would perfectly explain the missing data :(

More on this flag:
https://issues.collectionspace.org/browse/CSPACE-6322

AFAIK, the CollectionSpace documentation spotlights the destructive nature
of this command, wherever it's mentioned. E.g. in
https://wiki.collectionspace.org/display/DOC/Installing+4.3+on+Ubuntu+14.04:

[image: Inline image 1]
If the requisite user accounts are once again working, after this command
was run, reloading data might be the next step.

Aron

P.S. ***** Note to the project team: It looks like that warning needs to be
revised, however, to make it even clearer that all relevant databases -
that hold AuthN/AuthZ data, as well as all records and relationship data
pertinent to a CollectionSpace installation, across all tenants - are
dropped. That behavior might have been implicit in the current wording,
when it was originally written - back then, the 'cspace' database held auth
data and the 'nuxeo' database held all other data - but is no longer so.

On Thu, Jun 16, 2016 at 7:15 AM, James Keeline james@keeline.com wrote:

Hi James,

clean slate from which to work for this install.

Running 'ant undeploy deploy create_db -Drecreate_db=true import' once
again, from within the topmost directory of the source code tree for the
CollectionSpace Services layer, should do what you're asking for here, IIRC.

accounts_common.  However, the web UI will still not allow me to add the
user accounts.

Data connected with user accounts are present in multiple related tables in
CollectionSpace's AuthN/AuthZ database, beyond even those two mentioned
tables. So wiping everything, as mentioned above, is the way to go.

Aron

On Thu, Jun 16, 2016 at 9:54 AM, Aron Roberts aron@socrates.berkeley.edu
wrote:

I am coordinating with our sysadmin on this.
I just saw this.  He and I had been conferring and were going to restore a DB backup from a week or so ago.  This would be followed by a CSpace shutdown.sh -force and a startup.sh and probably the "initialise" and "init" URL calls after login.____
However, Aron, I see your recommendation that we run the command again from ~/cspace_source/services/
ant undeploy deploy create_db -Drecreate_db=true import
Thanks for jumping in to help again.  Once we have a stable install again, we can recreate data done manually and reload the person and object values via import services. 
James D. Keeline

  From: Aron Roberts <aron@socrates.berkeley.edu>

To: James Keeline james@keeline.com
Cc: Christopher R. HOFFMAN chris_h@berkeley.edu; Talk talk@lists.collectionspace.org
Sent: Thursday, June 16, 2016 9:58 AM
Subject: Re: [Talk] CSpace 4.3 -- data seems to have disappeared on production. In effort to replace a user, says UserID exists

Hi James,

Running 'ant undeploy deploy create_db -Drecreate_db=true import' once again, from within the topmost directory of the source code tree for the CollectionSpace Services layer, should do what you're asking for here, IIRC.

Data connected with user accounts are present in multiple related tables in CollectionSpace's AuthN/AuthZ database, beyond even those two mentioned tables. So wiping everything, as mentioned above, is the way to go.

Aron

On Thu, Jun 16, 2016 at 9:54 AM, Aron Roberts aron@socrates.berkeley.edu wrote:

Running the Ant 'create_db' target with the '-Drecreate_db=true' flag drops and re-creates databases, which would perfectly explain the missing data :(

More on this flag:
https://issues.collectionspace.org/browse/CSPACE-6322

AFAIK, the CollectionSpace documentation spotlights the destructive nature of this command, wherever it's mentioned. E.g. in https://wiki.collectionspace.org/display/DOC/Installing+4.3+on+Ubuntu+14.04:

If the requisite user accounts are once again working, after this command was run, reloading data might be the next step.

Aron

P.S. ***** Note to the project team: It looks like that warning needs to be revised, however, to make it even clearer that all relevant databases - that hold AuthN/AuthZ data, as well as all records and relationship data pertinent to a CollectionSpace installation, across all tenants - are dropped. That behavior might have been implicit in the current wording, when it was originally written - back then, the 'cspace' database held auth data and the 'nuxeo' database held all other data - but is no longer so. *****

On Thu, Jun 16, 2016 at 7:15 AM, James Keeline james@keeline.com wrote:

Seeing none of the data expected other than user accounts that don't show up in the web UI, I was going to get started on recreating the manual data.  I used psql to delete the two user account rows in accounts_tenants and accounts_common.  However, the web UI will still not allow me to add the user accounts.  This message is shown at the bottom of the browser screen:
POST failed: create in org.collectionspace.services.account.AccountResourceUserId exists. Non unique userId=tiffany@sdchm.org

So, before we get into the tedious work, I want to know how we may have a clean slate from which to work for this install.  This is our production server and we are previewing it to the director before next week's scheduled training.  On our side we'd like to get this done soon.
Thank you for the help so far. James D. Keeline

  From: James Keeline <james@keeline.com>

To: Christopher R. HOFFMAN chris_h@berkeley.edu; Aron Roberts aron@socrates.berkeley.edu
Cc: Talk talk@lists.collectionspace.org
Sent: Wednesday, June 15, 2016 9:50 PM
Subject: Re: [Talk] CSpace 4.3 -- data seems to have disappeared on production. In effort to replace a user, says UserID exists

That may indeed be the way to go.  I went through Aron's suggestions and the postgres tables are largely empty.  The exception is the accounts_common which did include the two additions even though not displayed in the web interface.
One can certainly spend too much time trying to understand what went wrong and/or fixing it when the best course of action may be to clear the databases and start again.
Is there a document that describes this process?  (Truncating or dropping tables or dropping entire databases and installing fresh).
We have most of the files (except my customizations to the form which I was installing this morning) in a git repository.  I would imagine that a fresh install of some kind will cause some confusion with that.
Aside from my installing files in the application and ui layer to customize the form, the other thing that our sysadmin did was assign a domain name to the IP.  He mentioned running an ant script as part of a process to fix the login issue.
ant undeploy deploy create_db -Drecreate_db=true import
Looking at the history, there were a number of commands that might purge the system.
However, I'm more interested in a path forward that will let us move towards deployment and training for the museum.
I appreciate the help and suggestions. James D. Keeline

  From: Christopher R. HOFFMAN <chris_h@berkeley.edu>

To: James Keeline james@keeline.com
Cc: Talk talk@lists.collectionspace.org
Sent: Wednesday, June 15, 2016 9:27 PM
Subject: Re: [Talk] CSpace 4.3 -- data seems to have disappeared on production. In effort to replace a user, says UserID exists

Very strange, James. I would recommend stepping back and then doing some things like checking the URLs that you are using. Maybe you are trying to log into the core tenant rather than your museum tenant? 
Otherwise yes I would say remove the database and move on.
Good luck, we are looking forward to hearing how it goes. 
Chris 
On Jun 15, 2016, at 4:42 PM, James Keeline james@keeline.com wrote:

I am bringing over the data from my local VirtualBox install over to our production system which was upgraded to 4.3.
On production I manually created:

• 2 user accounts for login
• 62 storage locations (described previously)
• 18 or so dynasties

I also used the import service to load the person records (artists, authors, employees).  That was done from an XML file and is repeatable easily when needed.
As we were preparing to preview it to the museum director before next week's training, I checked each of the above and found them present and pointed out what they should look for in terms of the customized form, select lists, and term lists.  It worked this morning.
This afternoon my PM said she could not log in as the director's account created and the data was missing.  When I visited myself, the site had many appearances of a new install.
I had resigned myself to having to create those manual values again, reimport the people, and this time make sure we had a DB backup for future restores.
After putting out some other fires, I began to create the first user account.  I was pretty surprised to see this error at the bottom:
POST failed: create in org.collectionspace.services.account.AccountResourceUserId exists. Non unique userId=tiffany@sdchm.org

So, this implies that my data is there and I might be able to restore it.  I'd appreciate suggestions on this.  
Sidebars of what can cause this are helpful too.
If I need to wipe the DB and start from scratch then I need that so we can go forward with recreating the data and loading the object data which a colleague has been working on (with some help from me).  He is having some success on his local install so I think we are making headway there.  There are some things I'm not sure about (CSpace object IDs to be assigned) but I need to focus on the above issue to meet our timeline.
Thank you again for any help or comments you can provide. James D. Keeline

Talk mailing list
Talk@lists.collectionspace.org
http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org

Talk mailing list
Talk@lists.collectionspace.org
http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org

My observation has been that if you're using instance-qualified databases, ant
create_db -Drecreate_db=true does not delete them. I think that's a bug,
since cspace now defaults to using instance-qualified databases. To start
fresh, you have to issue DROP DATABASE commands to delete all the cspace
databases. The names of the databases depend on the instance id you have
configured, but by default they'll be cspace_default, nuxeo_default,
fcart_default, etc. Given all the problems you're having, I would also
rebuild/redeploy the services, application, and ui layers.

Ray

On Thu, Jun 16, 2016 at 10:29 AM, James Keeline james@keeline.com wrote:

While our sysadmin is shepherding a fresh install into place, I have been making some more changes to the forms that we will use.
When we have the fresh install, I'd like to make the process of data once entered manually as efficient as possible.
As I've mentioned before, I'm repurposing Concepts >> Associated Concepts for our Chinese dynasty names.  There are 17 of these and they touch more than a half a dozen fields with the two languages, years, etc.
I have these in my local install.  Can I export them in some way and then reimport them (e.g. import service) to production?
If so, I will want to do the same with Storage Locations >> Local Storage Locations so advice on both is appreciated.  I have about 62 of these.
If they have to be done one by one then perhaps a Bash shell script can upload the individual XML files in a loop. James D. Keeline

  From: Ray Lee <rhlee@berkeley.edu>

To: James Keeline james@keeline.com
Cc: Aron Roberts aron@socrates.berkeley.edu; Talk talk@lists.collectionspace.org
Sent: Thursday, June 16, 2016 1:00 PM
Subject: Re: [Talk] CSpace 4.3 -- data seems to have disappeared on production. In effort to replace a user, says UserID exists

My observation has been that if you're using instance-qualified databases, ant create_db -Drecreate_db=true does not delete them. I think that's a bug, since cspace now defaults to using instance-qualified databases. To start fresh, you have to issue DROP DATABASE commands to delete all the cspace databases. The names of the databases depend on the instance id you have configured, but by default they'll be cspace_default, nuxeo_default, fcart_default, etc. Given all the problems you're having, I would also rebuild/redeploy the services, application, and ui layers.
Ray
On Thu, Jun 16, 2016 at 10:29 AM, James Keeline james@keeline.com wrote:

I am coordinating with our sysadmin on this.
I just saw this.  He and I had been conferring and were going to restore a DB backup from a week or so ago.  This would be followed by a CSpace shutdown.sh -force and a startup.sh and probably the "initialise" and "init" URL calls after login.____
However, Aron, I see your recommendation that we run the command again from ~/cspace_source/services/
ant undeploy deploy create_db -Drecreate_db=true import
Thanks for jumping in to help again.  Once we have a stable install again, we can recreate data done manually and reload the person and object values via import services. 
James D. Keeline

  From: Aron Roberts <aron@socrates.berkeley.edu>

To: James Keeline james@keeline.com
Cc: Christopher R. HOFFMAN chris_h@berkeley.edu; Talk talk@lists.collectionspace.org
Sent: Thursday, June 16, 2016 9:58 AM
Subject: Re: [Talk] CSpace 4.3 -- data seems to have disappeared on production. In effort to replace a user, says UserID exists

Hi James,

Running 'ant undeploy deploy create_db -Drecreate_db=true import' once again, from within the topmost directory of the source code tree for the CollectionSpace Services layer, should do what you're asking for here, IIRC.

Data connected with user accounts are present in multiple related tables in CollectionSpace's AuthN/AuthZ database, beyond even those two mentioned tables. So wiping everything, as mentioned above, is the way to go.

Aron

On Thu, Jun 16, 2016 at 9:54 AM, Aron Roberts aron@socrates.berkeley.edu wrote:

Running the Ant 'create_db' target with the '-Drecreate_db=true' flag drops and re-creates databases, which would perfectly explain the missing data :(

More on this flag:
https://issues.collectionspace.org/browse/CSPACE-6322

AFAIK, the CollectionSpace documentation spotlights the destructive nature of this command, wherever it's mentioned. E.g. in https://wiki.collectionspace.org/display/DOC/Installing+4.3+on+Ubuntu+14.04:

If the requisite user accounts are once again working, after this command was run, reloading data might be the next step.

Aron

P.S. ***** Note to the project team: It looks like that warning needs to be revised, however, to make it even clearer that all relevant databases - that hold AuthN/AuthZ data, as well as all records and relationship data pertinent to a CollectionSpace installation, across all tenants - are dropped. That behavior might have been implicit in the current wording, when it was originally written - back then, the 'cspace' database held auth data and the 'nuxeo' database held all other data - but is no longer so. *****

On Thu, Jun 16, 2016 at 7:15 AM, James Keeline james@keeline.com wrote:

Seeing none of the data expected other than user accounts that don't show up in the web UI, I was going to get started on recreating the manual data.  I used psql to delete the two user account rows in accounts_tenants and accounts_common.  However, the web UI will still not allow me to add the user accounts.  This message is shown at the bottom of the browser screen:
POST failed: create in org.collectionspace.services.account.AccountResourceUserId exists. Non unique userId=tiffany@sdchm.org

So, before we get into the tedious work, I want to know how we may have a clean slate from which to work for this install.  This is our production server and we are previewing it to the director before next week's scheduled training.  On our side we'd like to get this done soon.
Thank you for the help so far. James D. Keeline

  From: James Keeline <james@keeline.com>

To: Christopher R. HOFFMAN chris_h@berkeley.edu; Aron Roberts aron@socrates.berkeley.edu
Cc: Talk talk@lists.collectionspace.org
Sent: Wednesday, June 15, 2016 9:50 PM
Subject: Re: [Talk] CSpace 4.3 -- data seems to have disappeared on production. In effort to replace a user, says UserID exists

That may indeed be the way to go.  I went through Aron's suggestions and the postgres tables are largely empty.  The exception is the accounts_common which did include the two additions even though not displayed in the web interface.
One can certainly spend too much time trying to understand what went wrong and/or fixing it when the best course of action may be to clear the databases and start again.
Is there a document that describes this process?  (Truncating or dropping tables or dropping entire databases and installing fresh).
We have most of the files (except my customizations to the form which I was installing this morning) in a git repository.  I would imagine that a fresh install of some kind will cause some confusion with that.
Aside from my installing files in the application and ui layer to customize the form, the other thing that our sysadmin did was assign a domain name to the IP.  He mentioned running an ant script as part of a process to fix the login issue.
ant undeploy deploy create_db -Drecreate_db=true import
Looking at the history, there were a number of commands that might purge the system.
However, I'm more interested in a path forward that will let us move towards deployment and training for the museum.
I appreciate the help and suggestions. James D. Keeline

  From: Christopher R. HOFFMAN <chris_h@berkeley.edu>

To: James Keeline james@keeline.com
Cc: Talk talk@lists.collectionspace.org
Sent: Wednesday, June 15, 2016 9:27 PM
Subject: Re: [Talk] CSpace 4.3 -- data seems to have disappeared on production. In effort to replace a user, says UserID exists

Very strange, James. I would recommend stepping back and then doing some things like checking the URLs that you are using. Maybe you are trying to log into the core tenant rather than your museum tenant? 
Otherwise yes I would say remove the database and move on.
Good luck, we are looking forward to hearing how it goes. 
Chris 
On Jun 15, 2016, at 4:42 PM, James Keeline james@keeline.com wrote:

I am bringing over the data from my local VirtualBox install over to our production system which was upgraded to 4.3.
On production I manually created:

• 2 user accounts for login
• 62 storage locations (described previously)
• 18 or so dynasties

I also used the import service to load the person records (artists, authors, employees).  That was done from an XML file and is repeatable easily when needed.
As we were preparing to preview it to the museum director before next week's training, I checked each of the above and found them present and pointed out what they should look for in terms of the customized form, select lists, and term lists.  It worked this morning.
This afternoon my PM said she could not log in as the director's account created and the data was missing.  When I visited myself, the site had many appearances of a new install.
I had resigned myself to having to create those manual values again, reimport the people, and this time make sure we had a DB backup for future restores.
After putting out some other fires, I began to create the first user account.  I was pretty surprised to see this error at the bottom:
POST failed: create in org.collectionspace.services.account.AccountResourceUserId exists. Non unique userId=tiffany@sdchm.org

So, this implies that my data is there and I might be able to restore it.  I'd appreciate suggestions on this.  
Sidebars of what can cause this are helpful too.
If I need to wipe the DB and start from scratch then I need that so we can go forward with recreating the data and loading the object data which a colleague has been working on (with some help from me).  He is having some success on his local install so I think we are making headway there.  There are some things I'm not sure about (CSpace object IDs to be assigned) but I need to focus on the above issue to meet our timeline.
Thank you again for any help or comments you can provide. James D. Keeline

Talk mailing list
Talk@lists.collectionspace.org
http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org

Talk mailing list
Talk@lists.collectionspace.org
http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org

Talk mailing list
Talk@lists.collectionspace.org
http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org