Hi Susan and Jesse, I'm sending this out to the talk list as well, in case any other deployers are interested. I'll answer your questions in reverse order, since it's a bit easier to read that way. > What version of postgres are you using? We're (mostly by accident) using 9.0.4 in production, which we'll probably update to 9.1 when we get a chance. 9.1 is recommended by nuxeo. Our dev server is running 9.1, and it seems to perform slightly better, although it's unclear if the postgres version is the reason. Updating postgres from 9.0 to 9.1 requires dumping and restoring the database, so it's a bit of a pain if you have a lot of data. > I don't think we will be dropping the fulltext index or trigger for import > since they don't have so much data to migrate. Were there any changes to > the the distribution configuration for that in nuxeo's recommendations? We dropped the fulltext index, and the trigger that updates the fulltext column, following nuxeo's instructions. When we tried to recreate the index after the imports were complete, we found that nuxeo's instructions (http://doc.nuxeo.com/display/ADMINDOC55/Configuring+PostgreSQL#ConfiguringPostgreSQL-Massimportspecifictuning) for doing this didn't work. Instead of CREATE INDEX fulltext_fulltext_idx ON fulltext USING gin (fulltext); we used CREATE INDEX fulltext_fulltext_idx ON fulltext USING gin (nx_to_tsvector(fulltext::character varying)); > MMI is implementing 2.4 for their collection. Do you have a list of > patches that they really should add to the release version? I know there > are some critical patches to import that are needed and at least a few > others. Here are the commits that we've cherry-picked from master into our 2.4 branch, for each layer. This list may be missing a couple, because I think the git cherry command is not seeing the cherry-picks where there was a conflict. So I may revise this, after I figure out a better way. services: ray@Ray-Lee-C02GM5P6DV7P services (master)$ git cherry -v pahma_2.4 | egrep "^-.*? CSPACE" - e915c802358b3de4f1e703df53822d93c39465d4 CSPACE-5012: Simplified generation of collectionspace_core:uri values, based very loosely on a discussion of future registry directions with Patrick. inAuthority value in authority item uri values is currently stubbed. - c49102237ca5b9c438f68b5b7441107a13f9544a CSPACE-5012: Now appears to correctly add parent authority service and parent vocabulary CSID to collectionspace_core:uri for authority services. - ad772899457df97288e7e6d1e7855f85755e87dc CSPACE-5012: Made XPath expression evaluator a class variable; added multiple FIXME comments. - 44bf445919c9acc7637ed5725a7d608f9306d7e0 CSPACE-5012: Renamed method for generality; refined and added FIXME comments. - bf1d0d01b6aa6e7de9c6a042cc517c6d765ec646 CSPACE-5012: When populating collectionspace_core:uri with URIs for authority item records, handle extensions to the authorities doctypes by getting their base (unqualified) doctypes. - 473f65f7a2354b3b9a92a28eea98cf19ee1bae34 CSPACE-5012: Remove old comment. - ee1424f6ce60f8563e42580a5359c55e8299168d CSPACE-5012: More trivial comment cleanup. - d9e0d2cd809aebef96ca8cf531de765c9b78b4f9 CSPACE-5012: More minor cleanup. - ab46f95cedb031214e9d3064daa442c0c2f7ec56 CSPACE-5012: Renamed variables for greater clarity. - f9b0e688075dd6dcd19899c41786a139ce7d8c80 CSPACE-5012: Trivial clarifying change to method name. - 7ba23c6b17865bc8ac8e501558e48af0986422ec CSPACE-5012: Correct name of parent authority services for taxonomy records. - 3c275e50898d613526ae0c5b10c7c105f50c83f0 CSPACE-5229: Remove case-sensitivity when adding matching non-preferred terms to partial term search results. - f79fb3ea94d9fc90a232cbaa8cde36a136e449a9 CSPACE-5012: Use unqualified doctype in call to getServiceBindingForDocType. - 435a01dbaa330b5270abb9af317fac22e184c94a CSPACE-5012: Remove redundant code block following Ray's change to always remove tenant qualification from doctype names, when constructing the collectionspace_core:uri values for imported records. - a175ca56464548865ad102ff01e63612381b0ce5 CSPACE-5012: Tenant unqualify the docType prior to using it to construct the collectionspace_core:uri value; previous commit to TemplateExpander didn't do this. - 5c175b0b26dc10ad1ee6a5696030d263e5561c09 CSPACE-5012: Change inAuthority xpath to ignore namespace, and correct some doctypes. - d43220f86abfdeb48ba14c587b05efc5eba56c9d CSPACE-5012: Now correctly retrieves values from both namespaced and non-namespaced elements, as well as from records that have multiple schema parts (such as a common and extension part). Uses an abominable hack to wrap the XML fragment before extracting inAuthority value. - 76429bae7fc0c9537583605bd636cb5eb0339d53 CSPACE-5012: Remove some System.out messages added while debugging. - a2739d3d07408043447881852449e7481c7cd1a8 CSPACE-5292: updating Jasper iReports files to remove unused and incorrect accessiondate field that was recently converted into a structured date field. This new field type was breaking the Jasper template. Since that field, even when formatted correctly, wasn't used it was safe to remove entirely from the template. I also modified the report template in minor ways to fix field widths. - 46b62418de318ca2fe2153623792c2e411bab34f CSPACE-5241: Import records now accept additional attributes on the <import> tag: createdAt, createdBy, updatedAt, and updatedBy. These allow some implementer-requested metadata from legacy systems to be imported. - 33d1ff59494e0022ab9cf64e0dd552a04fcb96b9 CSPACE-5301: refName parsing for authorities (as contrasted with authority items) now also uses tokenization code exclusively, rather then tokenization code in some code path(s) and regex code in other(s). - 9793099a5cb106326363da3da8069b63f42bbc1e CSPACE-5301,CSPACE-5073: Added two unit test cases for refNamedisplay names containing colons and parens, both separator characters used elsewhere in refNames. - 3b20a1572ba9db74104692492d01b1088d52bf37 CSPACE-5301,CSPACE-5073: Minor changes to comments in refName test class. - b938805b4c01817bc382be3a72080dbef6464d31 CSPACE-4989, CSPACE-4115 Simplified queries that compute the referencing objects given a refName. Will improve performance, and make the system more stable. Enabled checking Cataloging records, for Used-By computations. - 15fc12187cee0f9df7c4278f1496788976e0428d CSPACE-3771, CSPACE-4791 refObjs call did not support wf_deleted filter, so was not correctly ignoring soft-deleted records. Since term delete checks for refObjs, this caused failure on term delete for which referencing objects had been (soft) deleted. - 28c5c9cf5ec90c8051ed4660613f62f777129ba4 CSPACE-4789: Call to start a new Nuxeo transaction now takes an optional timeout period paramter. application: ray@Ray-Lee-C02GM5P6DV7P application (master)$ git cherry -v pahma_2.4 | egrep "^-.*? CSPACE" - 12d515ba8e2e222a73ae94e472bb881ecbf0686c CSPACE-4978 - Adjusted VocabRedirector and WebAutoComplete to look for a "vocab" query param when considering hierarchy, and constrain the used vocabs to that if present. Ensures that hierarchy stays within a single vocabulary. UI passes the "namespace" for the record to use this. - 3e2f117c58657de4ed5f55ba110b10c35bd2f1b1 CSPACE-5192 Changed the way search on all vocabs works to be a single call to services, rather than a set of calls merged int he app layer. This is more efficient, and produces correct pagination behavior. - ff0d9af37a6cb3f6199e6c8ebe770a132345e9d5 CSPACE-5199 Fixed bug in the preparation of values for numeric data fields. - 79040eb4ea2e20a2bf653ca107b3453cd33a6eaa CSPACE-4228,PAHMA-374: Advanced search on text fields is once again case-insensitive. Formerly it was so only if wildcards were included in the search string. - e6fd2c116eeb037b637549ebc0fd9c9c6e57bb7e CSPACE-4228,PAHMA-374: Remove FIXME-type comment. - df18d2e1467375ed5f73d96b7280fcd3593fe87d CSPACE-4228,PAHMA-374: Remove additional comment. ui: ray@Ray-Lee-C02GM5P6DV7P ui (master)$ git cherry -v pahma_2.4 | egrep "^-.*? CSPACE" - 07421d6c24275370fbd992aaba1de55ffbf28fd8 CSPACE-3503: Make create new radio button labels enclose the radio buttons, so they're clickable. In addition, if you have structured date fields in extension schema, and those fields are repeating or in repeating groups, you should read CSPACE-5085, and you made need to apply the fix I made, in https://github.com/cspace-deployment/application/commit/d3e10f460d958798acf1ed95b1d312a6c02eb312. This fix is described in the jira. If you'd like to explore it more, the source code for the pahma deployment is in the pahma_2.4 branch in each of these repos: https://github.com/cspace-deployment/services https://github.com/cspace-deployment/application https://github.com/cspace-deployment/ui Thanks, Ray On Jul 5, 2012, at 9:43 AM, sstone@socrates.berkeley.edu wrote: > Hi Ray, > > MMI is implementing 2.4 for their collection. Do you have a list of > patches that they really should add to the release version? I know there > are some critical patches to import that are needed and at least a few > others. > > I don't think we will be dropping the fulltext index or trigger for import > since they don't have so much data to migrate. Were there any changes to > the the distribution configuration for that in nuxeo's recommendations? I > was confused by the last message Richard sent out. > > What version of postgres are you using? > > Thanks for your help. I understand if you are too busy with PAHMA to > respond for a while. > > Susan >

Ray, Thank you so much. You've done a lot of work for us all. Susan > Hi Susan and Jesse, > I'm sending this out to the talk list as well, in case any other deployers > are interested. I'll answer your questions in reverse order, since it's a > bit easier to read that way. > >> What version of postgres are you using? > > We're (mostly by accident) using 9.0.4 in production, which we'll probably > update to 9.1 when we get a chance. 9.1 is recommended by nuxeo. Our dev > server is running 9.1, and it seems to perform slightly better, although > it's unclear if the postgres version is the reason. Updating postgres from > 9.0 to 9.1 requires dumping and restoring the database, so it's a bit of a > pain if you have a lot of data. > >> I don't think we will be dropping the fulltext index or trigger for >> import >> since they don't have so much data to migrate. Were there any changes to >> the the distribution configuration for that in nuxeo's recommendations? > > > We dropped the fulltext index, and the trigger that updates the fulltext > column, following nuxeo's instructions. When we tried to recreate the > index after the imports were complete, we found that nuxeo's instructions > (http://doc.nuxeo.com/display/ADMINDOC55/Configuring+PostgreSQL#ConfiguringPostgreSQL-Massimportspecifictuning) > for doing this didn't work. Instead of > > CREATE INDEX fulltext_fulltext_idx ON fulltext USING gin (fulltext); > > we used > > CREATE INDEX fulltext_fulltext_idx ON fulltext USING gin > (nx_to_tsvector(fulltext::character varying)); > >> MMI is implementing 2.4 for their collection. Do you have a list of >> patches that they really should add to the release version? I know there >> are some critical patches to import that are needed and at least a few >> others. > > Here are the commits that we've cherry-picked from master into our 2.4 > branch, for each layer. This list may be missing a couple, because I think > the git cherry command is not seeing the cherry-picks where there was a > conflict. So I may revise this, after I figure out a better way. > > services: > > ray@Ray-Lee-C02GM5P6DV7P services (master)$ git cherry -v pahma_2.4 | > egrep "^-.*? CSPACE" > - e915c802358b3de4f1e703df53822d93c39465d4 CSPACE-5012: Simplified > generation of collectionspace_core:uri values, based very loosely on a > discussion of future registry directions with Patrick. inAuthority value > in authority item uri values is currently stubbed. > - c49102237ca5b9c438f68b5b7441107a13f9544a CSPACE-5012: Now appears to > correctly add parent authority service and parent vocabulary CSID to > collectionspace_core:uri for authority services. > - ad772899457df97288e7e6d1e7855f85755e87dc CSPACE-5012: Made XPath > expression evaluator a class variable; added multiple FIXME comments. > - 44bf445919c9acc7637ed5725a7d608f9306d7e0 CSPACE-5012: Renamed method for > generality; refined and added FIXME comments. > - bf1d0d01b6aa6e7de9c6a042cc517c6d765ec646 CSPACE-5012: When populating > collectionspace_core:uri with URIs for authority item records, handle > extensions to the authorities doctypes by getting their base (unqualified) > doctypes. > - 473f65f7a2354b3b9a92a28eea98cf19ee1bae34 CSPACE-5012: Remove old > comment. > - ee1424f6ce60f8563e42580a5359c55e8299168d CSPACE-5012: More trivial > comment cleanup. > - d9e0d2cd809aebef96ca8cf531de765c9b78b4f9 CSPACE-5012: More minor > cleanup. > - ab46f95cedb031214e9d3064daa442c0c2f7ec56 CSPACE-5012: Renamed variables > for greater clarity. > - f9b0e688075dd6dcd19899c41786a139ce7d8c80 CSPACE-5012: Trivial clarifying > change to method name. > - 7ba23c6b17865bc8ac8e501558e48af0986422ec CSPACE-5012: Correct name of > parent authority services for taxonomy records. > - 3c275e50898d613526ae0c5b10c7c105f50c83f0 CSPACE-5229: Remove > case-sensitivity when adding matching non-preferred terms to partial term > search results. > - f79fb3ea94d9fc90a232cbaa8cde36a136e449a9 CSPACE-5012: Use unqualified > doctype in call to getServiceBindingForDocType. > - 435a01dbaa330b5270abb9af317fac22e184c94a CSPACE-5012: Remove redundant > code block following Ray's change to always remove tenant qualification > from doctype names, when constructing the collectionspace_core:uri values > for imported records. > - a175ca56464548865ad102ff01e63612381b0ce5 CSPACE-5012: Tenant unqualify > the docType prior to using it to construct the collectionspace_core:uri > value; previous commit to TemplateExpander didn't do this. > - 5c175b0b26dc10ad1ee6a5696030d263e5561c09 CSPACE-5012: Change inAuthority > xpath to ignore namespace, and correct some doctypes. > - d43220f86abfdeb48ba14c587b05efc5eba56c9d CSPACE-5012: Now correctly > retrieves values from both namespaced and non-namespaced elements, as well > as from records that have multiple schema parts (such as a common and > extension part). Uses an abominable hack to wrap the XML fragment before > extracting inAuthority value. > - 76429bae7fc0c9537583605bd636cb5eb0339d53 CSPACE-5012: Remove some > System.out messages added while debugging. > - a2739d3d07408043447881852449e7481c7cd1a8 CSPACE-5292: updating Jasper > iReports files to remove unused and incorrect accessiondate field that was > recently converted into a structured date field. This new field type was > breaking the Jasper template. Since that field, even when formatted > correctly, wasn't used it was safe to remove entirely from the template. I > also modified the report template in minor ways to fix field widths. > - 46b62418de318ca2fe2153623792c2e411bab34f CSPACE-5241: Import records now > accept additional attributes on the <import> tag: createdAt, createdBy, > updatedAt, and updatedBy. These allow some implementer-requested metadata > from legacy systems to be imported. > - 33d1ff59494e0022ab9cf64e0dd552a04fcb96b9 CSPACE-5301: refName parsing > for authorities (as contrasted with authority items) now also uses > tokenization code exclusively, rather then tokenization code in some code > path(s) and regex code in other(s). > - 9793099a5cb106326363da3da8069b63f42bbc1e CSPACE-5301,CSPACE-5073: Added > two unit test cases for refNamedisplay names containing colons and parens, > both separator characters used elsewhere in refNames. > - 3b20a1572ba9db74104692492d01b1088d52bf37 CSPACE-5301,CSPACE-5073: Minor > changes to comments in refName test class. > - b938805b4c01817bc382be3a72080dbef6464d31 CSPACE-4989, CSPACE-4115 > Simplified queries that compute the referencing objects given a refName. > Will improve performance, and make the system more stable. Enabled > checking Cataloging records, for Used-By computations. > - 15fc12187cee0f9df7c4278f1496788976e0428d CSPACE-3771, CSPACE-4791 > refObjs call did not support wf_deleted filter, so was not correctly > ignoring soft-deleted records. Since term delete checks for refObjs, this > caused failure on term delete for which referencing objects had been > (soft) deleted. > - 28c5c9cf5ec90c8051ed4660613f62f777129ba4 CSPACE-4789: Call to start a > new Nuxeo transaction now takes an optional timeout period paramter. > > > application: > > ray@Ray-Lee-C02GM5P6DV7P application (master)$ git cherry -v pahma_2.4 | > egrep "^-.*? CSPACE" > - 12d515ba8e2e222a73ae94e472bb881ecbf0686c CSPACE-4978 - Adjusted > VocabRedirector and WebAutoComplete to look for a "vocab" query param when > considering hierarchy, and constrain the used vocabs to that if present. > Ensures that hierarchy stays within a single vocabulary. UI passes the > "namespace" for the record to use this. > - 3e2f117c58657de4ed5f55ba110b10c35bd2f1b1 CSPACE-5192 Changed the way > search on all vocabs works to be a single call to services, rather than a > set of calls merged int he app layer. This is more efficient, and produces > correct pagination behavior. > - ff0d9af37a6cb3f6199e6c8ebe770a132345e9d5 CSPACE-5199 Fixed bug in the > preparation of values for numeric data fields. > - 79040eb4ea2e20a2bf653ca107b3453cd33a6eaa CSPACE-4228,PAHMA-374: Advanced > search on text fields is once again case-insensitive. Formerly it was so > only if wildcards were included in the search string. > - e6fd2c116eeb037b637549ebc0fd9c9c6e57bb7e CSPACE-4228,PAHMA-374: Remove > FIXME-type comment. > - df18d2e1467375ed5f73d96b7280fcd3593fe87d CSPACE-4228,PAHMA-374: Remove > additional comment. > > > ui: > > ray@Ray-Lee-C02GM5P6DV7P ui (master)$ git cherry -v pahma_2.4 | egrep > "^-.*? CSPACE" > - 07421d6c24275370fbd992aaba1de55ffbf28fd8 CSPACE-3503: Make create new > radio button labels enclose the radio buttons, so they're clickable. > > > In addition, if you have structured date fields in extension schema, and > those fields are repeating or in repeating groups, you should read > CSPACE-5085, and you made need to apply the fix I made, in > https://github.com/cspace-deployment/application/commit/d3e10f460d958798acf1ed95b1d312a6c02eb312. > This fix is described in the jira. > > If you'd like to explore it more, the source code for the pahma deployment > is in the pahma_2.4 branch in each of these repos: > > https://github.com/cspace-deployment/services > https://github.com/cspace-deployment/application > https://github.com/cspace-deployment/ui > > Thanks, > Ray > > > On Jul 5, 2012, at 9:43 AM, sstone@socrates.berkeley.edu wrote: > >> Hi Ray, >> >> MMI is implementing 2.4 for their collection. Do you have a list of >> patches that they really should add to the release version? I know there >> are some critical patches to import that are needed and at least a few >> others. >> >> I don't think we will be dropping the fulltext index or trigger for >> import >> since they don't have so much data to migrate. Were there any changes to >> the the distribution configuration for that in nuxeo's recommendations? >> I >> was confused by the last message Richard sent out. >> >> What version of postgres are you using? >> >> Thanks for your help. I understand if you are too busy with PAHMA to >> respond for a while. >> >> Susan >> > >