Hi Susan,

So far I've only used Talend for generating CollectionObjects (no repeated fields, relations or any other schemas yet). The AdvancedXML output module does not seem to directly support multiple repeated fields but I think it's possible to find a way around this limitation (by formatting them before sending to the module). Talend can be configured to generate complete xml files (one for each object) ready for import to nuxeo (see attachment).

Post processing is limited to this: To satisfy the nuxeo import format, each generated xml file is placed in it's own unique directory and renamed (to document.xml). This directory path (relative to the nuxeo directory tree) is inserted into the xml document in the <document><system><path> part of the xml.

CSIDs: The final node of the <path> (see above) becomes the CSID (so is in fact whatever name I've chosen to give the directory directly holding the xml file). At the moment, I'm not using particularly well thought out csid's (not UUIDs).

But I'm glad you asked this because it makes me think.... Is it a requirement that Services generates the CSID? In fact it would be interesting to hear from someone whether is it a requirement to use Services APIs to migrate data in general, or is nuxeo shell also an approved approach?

If it proves necessary to avoid nuxeo shell I think it would be optimal to access the service APIs directly from the ETL (requires building a new java component in Talend). Otherwise, I guess your java client could be adapted to handle the Talend (xml) output instead.

Regards,
Chris

Developer, Corpus Project
Statens Museum for Kunst (National Gallery of Denmark)

-----Oprindelig meddelelse-----
Fra: Susan Stone [mailto:sstone@socrates.berkeley.edu]
Sendt: 1. juli 2010 20:08
Til: Christopher Pott
Cc: Richard Millet; Glen Jackson; Chris Hoffman
Emne: Re: [Talk] Deployment experience at SMK?

Chris,

I'm working on data migration for the Hearst Museum at Berkeley, and I
guess you've seen some of the documentation of our project that Chris
Hoffman has put up. So far I am using the Pentaho (Kettle) tool to
create text files that I load using the CSpace services java client API.
I expect I am going to need to start creating XML when it becomes
possible to load "repeating" fields and more complex field groupings,
and I'm not sure if Talend will be better than Kettle for this.

Could you send me a sample of the "advanced XML" output for loading into
CSpace that you get out of the Talend ETL tool (before you do further
manipulation)? Does it output a series of XML records in a single file,
multiple files, or are you able to pipe (or something) output from
Talend to the Nuxeo shell that you are using?

Also, how are you creating Collection Space IDs (csids) in your load
process?

Thanks,
Susan Stone
Informatics Group, Data Services, UC Berkeley

Christopher Pott wrote:

This is great info, Chris. Thanks for sharing it.

On a separate thread of work, I am exploring how we can do bulk import to
Nuxeo, creating both our CSIDs as well as their internal UUID values.
Initially, I will be doing this for bulk import of vocabularies and
authorities, but what we learn should of course apply to import in general.

I'll keep the list posted on progress.

Thanks - Patrick

Thanks so much, Chris. A very different approach from what I'm doing so
far. Let's continue to exchange notes.

I hope the services group will be able to answer your questions on what
data loading methods are likely to be OK.

I may be trying out various things using the CSpace java and REST client
APIs with XML payloads (with guidance from the services group). Kettle
will let me run shell script jobs using the output of a transformation,
and I'll try using that before I think about a CSpace plug-in. I'll also
take a look at Talend and see what it offers.

Since CSpace is generating csids for me, it is important that I am able
to get them back so that I can use them later in relations and in some
cases updates (I'm doing some of my loading as updates). So part of my
approach is associating generated csids with IDs in the original data.

Thanks again for describing your approach.

Susan

Christopher Pott wrote:

Hi Chris,

At 14:35 +0200 2010-07-02, Christopher Pott wrote:

My sense - and this is personal opinion only -
is that externally-assigned CSIDs for records in
CollectionSpace should ideally be UUIDs.  There
is additional merit in using Type 4 pseudorandom
UUIDs, as generated by Java's UUID.randomUUID()
method or comparable code in other programming or
scripting languages, which match the identifiers
assigned to records by the CollectionSpace
services.

Doing so:

• Helps ensure uniqueness - at least to an extraordinarily high
  probability.

http://en.wikipedia.org/wiki/Universally_Unique_Identifier#Random_UUID_probability_of_duplicates

• Invokes code in core Java, with a (presumably) well-designed
  and tested algorithm - not to mention a robust code maintenance
  infrastructure - to generate identifiers.

• Standardizes on a single format that can be relied upon when
  and if it may be necessary to validate CSIDs (e.g. see regex below).

Using other types of identifiers may not offer the above benefits.

Aron

P.S. These UUIDs are simple to generate, as in this Java utility class:

import java.util.UUID;
public class GenerateCSID {
public static void main( String args[] ) {
UUID id = UUID.randomUUID();
System.out.println( id.toString() );
}
}

The generated CSIDs can also be matched by this regex:

([a-z0-9-]{8}-[a-z0-9-]{4}-4[a-z0-9-]{3}-[89ab][a-z0-9-]{3}-[a-z0-9-]{12})

--

Date: Fri, 2 Jul 2010 14:35:15 +0200
From: "Christopher Pott" Christopher.Pott@smk.dk
To: "Susan Stone" sstone@socrates.berkeley.edu
Cc: Glen Jackson glen@berkeley.edu, talk@lists.collectionspace.org
Subject: Re: [Talk] Deployment experience at SMK?

Hi Susan,

So far I've only used Talend for generating
CollectionObjects (no repeated fields, relations
or any other schemas yet). The AdvancedXML output
module does not seem to directly support multiple
repeated fields but I think it's possible to find
a way around this limitation (by formatting them
before sending to the module). Talend can be
configured to generate complete xml files (one
for each object) ready for import to nuxeo (see
attachment).

Post processing is limited to this: To satisfy
the nuxeo import format, each generated xml file
is placed in it's own unique directory and
renamed (to document.xml). This directory path
(relative to the nuxeo directory tree) is
inserted into the xml document in the
<document><system><path> part of the xml.

CSIDs: The final node of the <path> (see above)
becomes the CSID (so is in fact whatever name
I've chosen to give the directory directly
holding the xml file). At the moment, I'm not
using particularly well thought out csid's (not
UUIDs).

But I'm glad you asked this because it makes me
think.... Is it a requirement that Services
generates the CSID? In fact it would be
interesting to hear from someone whether is it a
requirement to use Services APIs to migrate data
in general, or is nuxeo shell also an approved
approach?

If it proves necessary to avoid nuxeo shell I
think it would be optimal to access the service
APIs directly from the ETL (requires building a
new java component in Talend). Otherwise, I guess
your java client could be adapted to handle the
Talend (xml) output instead.

Regards,
Chris

Developer, Corpus Project
Statens Museum for Kunst (National Gallery of Denmark)

-----Oprindelig meddelelse-----
Fra: Susan Stone [mailto:sstone@socrates.berkeley.edu]
Sendt: 1. juli 2010 20:08
Til: Christopher Pott
Cc: Richard Millet; Glen Jackson; Chris Hoffman
Emne: Re: [Talk] Deployment experience at SMK?

Chris,

I'm working on data migration for the Hearst Museum at Berkeley, and I
guess you've seen some of the documentation of our project that Chris
Hoffman has put up. So far I am using the Pentaho (Kettle) tool to
create text files that I load using the CSpace services java client API.
I expect I am going to need to start creating XML when it becomes
possible to load "repeating" fields and more complex field groupings,
and I'm not sure if Talend will be better than Kettle for this.

Could you send me a sample of the "advanced XML" output for loading into
CSpace that you get out of the Talend ETL tool (before you do further
manipulation)? Does it output a series of XML records in a single file,
multiple files, or are you able to pipe (or something) output from
Talend to the Nuxeo shell that you are using?

Also, how are you creating Collection Space IDs (csids) in your load
process?

Thanks,
Susan Stone
Informatics Group, Data Services, UC Berkeley

Christopher Pott wrote:

At 12:10 -0700 2010-07-02, Aron Roberts wrote:

And a fourth benefit: helps ensure consistency of identifiers with
those assigned in any future bulk import mechanism in
CollectionSpace, such as that which Patrick is currently exploring:

At 09:29 -0700 2010-07-02, Patrick Schmitz wrote:

Aron

I have an idea that you might need to initialise the controlled lists

if you ran the tests on mvn  for the app layer then it should have done
it for you. But if you skipped the tests you can run the initialization
from a browser

first go to
/cspace-ui/html and login
then go to
/chain/authorities/vocab/initialize

unfortunately I haven't set it up to give any useful messages on the
screen but it should only take a moment

then log back in and see if the pages work.

if you are missing the default person and org authorities (and if you
need them)
you can get them by going to
/chain/reset
(once again you must have logged in to the ui or else the authorization
will fail)

Tell me how it goes

Chris M

On 12/07/2010 15:32, Christopher Pott wrote:

Hi Chris,

In my initial attempt I downloaded the chain.war from the
collectionspace ftp server, so I had no interaction with mvn in regards
to the application layer. Maybe there is a problem with this war?

The good news is that '/chain/authorities/vocab/initialize' did the
trick for making the missing templates visible (although it returned a
400 error). However, there is now quite a noticible delay (10-20
seconds) when the UI attemps to GET /chain/objects/uispec.

I also tried chain/reset which worked fine, but I think was not needed
in my case as I will in case be populating these objects with our own
data.

Thanks for the help,
Chris

-----Oprindelig meddelelse-----
Fra: Chris Martin [mailto:csm22@caret.cam.ac.uk]
Sendt: 12. juli 2010 16:52
Til: Christopher Pott
Cc: talk@lists.collectionspace.org
Emne: Re: [Talk] problem running v0.8

I have an idea that you might need to initialise the controlled lists

if you ran the tests on mvn  for the app layer then it should have done
it for you. But if you skipped the tests you can run the initialization
from a browser

first go to
/cspace-ui/html and login
then go to
/chain/authorities/vocab/initialize

unfortunately I haven't set it up to give any useful messages on the
screen but it should only take a moment

then log back in and see if the pages work.

if you are missing the default person and org authorities (and if you
need them)
you can get them by going to
/chain/reset
(once again you must have logged in to the ui or else the authorization
will fail)

Tell me how it goes

Chris M

On 12/07/2010 15:32, Christopher Pott wrote:

go

a

by

is

checking

any

http://lists.collectionspace.org/mailman/listinfo/talk_lists.collections
pace.org

On 13/07/2010 13:09, Christopher Pott wrote:

no the war is fine.
However, if you want to auto populate data like the controlled
vocabulariezed lists then script need to be run when the war is in place
and can talk to the service layer.
We need to update the documentation to add this stage - sorry for the
confusion

Yeap the time take for objects/uispec is a known issue and should be
resolved in v1.0a as controlled lists will be handled differently so
should need to worry the uispec as much