talk@lists.collectionspace.org

WE HAVE SUNSET THIS LISTSERV - Join us at collectionspace@lyrasislists.org

View all threads

REST update for large number of records

CH
Chris Hoffman
Wed, May 16, 2012 5:42 PM

OK, here's a question!  We need to run some REST updates for many, many records (30K records need a collectionspace_core:uri value).  We are creating the payloads in Talend, but are trying to figure out how to script this.  If we use curl to send the PUT commands, we think we need to create an individual xml payload file for each record.  This is possible but clunky.

Is there a way to do a PUT by parsing through one big file (output from Talend) and sending the payloads for the individual "records" in more of a stream mode?  I hope this makes some sense.

Thanks,
Chris

OK, here's a question! We need to run some REST updates for many, many records (30K records need a collectionspace_core:uri value). We are creating the payloads in Talend, but are trying to figure out how to script this. If we use curl to send the PUT commands, we think we need to create an individual xml payload file for each record. This is possible but clunky. Is there a way to do a PUT by parsing through one big file (output from Talend) and sending the payloads for the individual "records" in more of a stream mode? I hope this makes some sense. Thanks, Chris
AR
Aron Roberts
Wed, May 16, 2012 7:45 PM

That sounds eminently scriptable ... each PUT request will still need
a single payload, but that payload can be parsed out of a larger XML
file, wrapped with whatever headers are required, etc.

On Wed, May 16, 2012 at 10:42 AM, Chris Hoffman
chris.hoffman@berkeley.edu wrote:

OK, here's a question!  We need to run some REST updates for many, many records (30K records need a collectionspace_core:uri value).  We are creating the payloads in Talend, but are trying to figure out how to script this.  If we use curl to send the PUT commands, we think we need to create an individual xml payload file for each record.  This is possible but clunky.

Is there a way to do a PUT by parsing through one big file (output from Talend) and sending the payloads for the individual "records" in more of a stream mode?  I hope this makes some sense.

Thanks,
Chris


Talk mailing list
Talk@lists.collectionspace.org
http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org

That sounds eminently scriptable ... each PUT request will still need a single payload, but that payload can be parsed out of a larger XML file, wrapped with whatever headers are required, etc. On Wed, May 16, 2012 at 10:42 AM, Chris Hoffman <chris.hoffman@berkeley.edu> wrote: > OK, here's a question!  We need to run some REST updates for many, many records (30K records need a collectionspace_core:uri value).  We are creating the payloads in Talend, but are trying to figure out how to script this.  If we use curl to send the PUT commands, we think we need to create an individual xml payload file for each record.  This is possible but clunky. > > Is there a way to do a PUT by parsing through one big file (output from Talend) and sending the payloads for the individual "records" in more of a stream mode?  I hope this makes some sense. > > Thanks, > Chris > > > _______________________________________________ > Talk mailing list > Talk@lists.collectionspace.org > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org