AS
Angela Spinazze
Mon, Apr 23, 2012 2:33 PM
Hi all and thanks for the discussion about 2.4. I have cc'd the Talk list so that everyone can see this discussion as it is precisely the type of conversation that we want to have happen in our community space.
What I take away from the discussion thread is that there is a tension between taking on the merging work now versus shifting that work to a later point in our development schedule so that the immediate needs of deployments at UCB might instead be more efficiently addressed (based on current resource constraints). The merging work has been talked about and pushed out for over a year now so the landscape looks different from the deployer perspective than it once did, yet the change to the model that the merging work represents remains critical to the long-term viability of the software for future deployers.
A couple of questions that come to mind are:
What do we lose by not doing the merging work now?
I think that the answer is (based on Patrick's comments below) that we lose the ability to extend repeatable groups, and we lose support for localization (neither of which we have now). Is that correct? Chris, you pointed out that UCB has already addressed repeatable groups through a work-around. Chris Pott has confirmed that he has had to do the same thing at SMK and I suspect that MMI and Walker have had to as well (but I don't know for certain- Jesse, Chris, Nate please chime in if you are so moved). This means that while the merging work will make extending repeatable groups easier, more efficient and effective in the future, current deployers won't necessarily benefit from the change to the model right now. Same goes for localization.
The other work defined for 2.4 ensures that we gain support for preferred and non-preferred terms and support for non-default vocabularies and is defined as a low-risk, achievable set of deliverables for a 2-week development sprint.
What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we do not do the merging work now?
I'm not sure that I know what the answer is to this question. Would we, for example, still be able to work on relationships between catalog records (AKA multiple object handling) in 2.7 without the merging work? Is anything related to media handling(2.6) or improving the upgrade process (2.6) affected by not doing the merging work?
If what we gain from pushing the merging work out to a later time in the development schedule is more beneficial for all current deployers, then I support the shift. It sounds like that is the case, but I need your help Patrick with answers to the questions above, in order to fully understand the ramifications.
Angela
On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
Yes, this is all correct so far.
We could take a messier approach with the PT/NPT changes. That would work, but is a bit messier in a number of places. However, the messier approach is well understood and low risk.
We had been planning to take an approach that was cleaner, but required support for merging.
Yesterday, discussions about how we would actually implement the merging led to some questions about how this would look in the database, and the implications for report authoring. There was some surprise from our deployers at how complex the so-called "merged" model would be in the DB.
For SMK, it will mean that any report authoring will be more complex if they are injecting language all over.
Richard and I are feeling ambivalent about this. As software engineers, we feel that the functionality should be there to support merging, and that it would make the code for services and app cleaner, and would present a nice model to the UI. However, being practical, and considering the additional pain that this will entail for reporting, I am having some doubts. Knowing how long it is taking me to get code working in the app layer, I am concerned about the risk.
The details:
Let's say we want to add language to some repeating group of fields (let's call it FooInfo). This will entail setting up specific fields for language in an extension schema (it actually has to be something like smk_FooInfoList/smk_FooInfo/language), some associated config in the app layer, and the obvious UI work to present it to the user.
What happens on the back-end is that the new language fields end up in a separate table (smk_FooInfo) that the app layer and services have to keep aligned with the FooInfo part of the common schema. The UI and app layer should be able to ensure that if there are three rows in FooInfo, there will be three rows in smk_FooInfo. This is the code I would have to write, and it entails changes to the core of the app layer, so is very high risk.
Issues:
If someone uses the REST APIs or import and does not get this right, everything will get hosed (e.g., SMK's language markers will end up on the wrong row for the notes they were supposed to be associated with. There is no reasonable way for services to even detect when an import or REST call has screwed this up.
When someone wants to generate a report that includes the language (or selects for the language) for a note, the query will get a little funky, requiring more careful joins and constraints in what is already a complex data model.
ChrisH was surprised that the fields would not be merged into the common table, until we explained that it cannot without changing the common model in the code and for all the users. Even if we split up the repository to be one per tenant, our code (a lot of the the REST stuff) would get confused by not having a consistent definition of the schema, and so we have to add a new table for every injected field.
One alternative is to defer or just not support merging. Instead, we would either:
Tell folks to live with the model they have now, in which they have to redefine the whole repeating structure everywhere they want to add a field. Ugly, but well understood. The main objection to this after the amount of typing, was that upgrades could be problematic. If we can stabilize the schema and stop changing them so much (which we really need to do anyway), this should not be so much of an issue.
OR
Be more liberal about adding new fields where folks need them, even if we are not supporting them in the UI right away. E.g., adding Language to a bunch of groups would cause us little pain, and then SMK could contribute templates that use the language fields.
Neither of these are great, and I do not mean to be glib in suggesting them. Am trying to be practical and realistic about our current resource situation.
HTH - Patrick
From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
Sent: Friday, April 20, 2012 9:01 AM
To: Carly Bogen
Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM; Patrick Schmitz
Subject: Re: Concerns about 2.4
Thanks, Carly. Yes, you're right. Patrick has been calling this work "merging" because the app layer needs to merge the extension fields into the core group somehow. This would be needed for PT/NPT if the common set of term fields were all in one shared schema. However, Patrick has acknowledged that the "common" fields could be in their own authority-specific schema. We just have extra work to try to make sure those fields don't diverge in the future. I might not be translating this correctly :-)
Chris
On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
Hi Chris,
Based on the design and scope discussions we had this week, preferred and non-preferred terms and support for non-default vocabularies are the main drivers for 2.4. I'm not sure where the concern about extending core field groups came from, since it was not something we discussed. Is this related to the merging issue? If so, it was my understanding that fixing the merging issue was key to making PT and NPT and non-default vocabularies work.
Patrick, maybe you can weigh in on this and help me understand.
Thanks,
Carly
On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman chris.hoffman@berkeley.edu wrote:
Hi all,
I'm really, really happy to see how much work the team has done to make 2.4 more realistic. However at UCB we still have some concerns that I want to bring up with you and other deployers.
In talking with Patrick and Richard yesterday, I'm concerned that the work to enable deployers to extend core field groups is going to swamp other areas of work that for us are absolutely critical. PAHMA needs to launch with version 2.4, or they will have to extend TMS licensing (probably costing tens of thousands of dollars) and answer to campus auditors.
We also want to add fields to existing groups, but in the meantime, we have created our own versions of those groups as needed. The work Patrick described to me is hard, risky, and has other implications for data loading and reporting (reporting will be very hard and the approach might have to change completely to point iReport at the App layer instead of at Postgres tables). Patrick is the only one who can do this work now, and he is concerned about being on the critical path for something like this. He is off for 3 weeks starting in early June.
For PAHMA to launch with 2.4, they need the preferred/non-preferred term stuff, the support for multiple vocabularies within a single authority, and a fix to the dollar-sign import bug. There is also some significant customization work here that will need help from Patrick's team (e.g., supporting range search on object numbers, "show me objects with numbers 91-202 through 91-292").
I suspect field group extension is critical for SMK in particular. However, we really are on the edge of a major situation here.
Thanks,
Chris
Hi all and thanks for the discussion about 2.4. I have cc'd the Talk list so that everyone can see this discussion as it is precisely the type of conversation that we want to have happen in our community space.
What I take away from the discussion thread is that there is a tension between taking on the merging work now versus shifting that work to a later point in our development schedule so that the immediate needs of deployments at UCB might instead be more efficiently addressed (based on current resource constraints). The merging work has been talked about and pushed out for over a year now so the landscape looks different from the deployer perspective than it once did, yet the change to the model that the merging work represents remains critical to the long-term viability of the software for future deployers.
A couple of questions that come to mind are:
What do we lose by not doing the merging work now?
I think that the answer is (based on Patrick's comments below) that we lose the ability to extend repeatable groups, and we lose support for localization (neither of which we have now). Is that correct? Chris, you pointed out that UCB has already addressed repeatable groups through a work-around. Chris Pott has confirmed that he has had to do the same thing at SMK and I suspect that MMI and Walker have had to as well (but I don't know for certain- Jesse, Chris, Nate please chime in if you are so moved). This means that while the merging work will make extending repeatable groups easier, more efficient and effective in the future, current deployers won't necessarily benefit from the change to the model right now. Same goes for localization.
The other work defined for 2.4 ensures that we gain support for preferred and non-preferred terms and support for non-default vocabularies and is defined as a low-risk, achievable set of deliverables for a 2-week development sprint.
What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we do not do the merging work now?
I'm not sure that I know what the answer is to this question. Would we, for example, still be able to work on relationships between catalog records (AKA multiple object handling) in 2.7 without the merging work? Is anything related to media handling(2.6) or improving the upgrade process (2.6) affected by not doing the merging work?
If what we gain from pushing the merging work out to a later time in the development schedule is more beneficial for all current deployers, then I support the shift. It sounds like that is the case, but I need your help Patrick with answers to the questions above, in order to fully understand the ramifications.
Angela
On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
> Yes, this is all correct so far.
>
> We *could* take a messier approach with the PT/NPT changes. That would work, but is a bit messier in a number of places. However, the messier approach is well understood and low risk.
>
> We had been planning to take an approach that was cleaner, but required support for merging.
>
> Yesterday, discussions about how we would actually implement the merging led to some questions about how this would look in the database, and the implications for report authoring. There was some surprise from our deployers at how complex the so-called "merged" model would be in the DB.
>
> For SMK, it will mean that any report authoring will be more complex if they are injecting language all over.
>
> Richard and I are feeling ambivalent about this. As software engineers, we feel that the functionality should be there to support merging, and that it would make the code for services and app cleaner, and would present a nice model to the UI. However, being practical, and considering the additional pain that this will entail for reporting, I am having some doubts. Knowing how long it is taking me to get code working in the app layer, I am concerned about the risk.
>
> The details:
>
> Let's say we want to add language to some repeating group of fields (let's call it FooInfo). This will entail setting up specific fields for language in an extension schema (it actually has to be something like smk_FooInfoList/smk_FooInfo/language), some associated config in the app layer, and the obvious UI work to present it to the user.
>
> What happens on the back-end is that the new language fields end up in a separate table (smk_FooInfo) that the app layer and services have to keep aligned with the FooInfo part of the common schema. The UI and app layer should be able to ensure that if there are three rows in FooInfo, there will be three rows in smk_FooInfo. This is the code I would have to write, and it entails changes to the core of the app layer, so is very high risk.
>
> Issues:
> If someone uses the REST APIs or import and does not get this right, everything will get hosed (e.g., SMK's language markers will end up on the wrong row for the notes they were supposed to be associated with. There is no reasonable way for services to even detect when an import or REST call has screwed this up.
> When someone wants to generate a report that includes the language (or selects for the language) for a note, the query will get a little funky, requiring more careful joins and constraints in what is already a complex data model.
> ChrisH was surprised that the fields would not be merged into the common table, until we explained that it cannot without changing the common model in the code and for all the users. Even if we split up the repository to be one per tenant, our code (a lot of the the REST stuff) would get confused by not having a consistent definition of the schema, and so we have to add a new table for every injected field.
>
> One alternative is to defer or just not support merging. Instead, we would either:
> Tell folks to live with the model they have now, in which they have to redefine the whole repeating structure everywhere they want to add a field. Ugly, but well understood. The main objection to this after the amount of typing, was that upgrades could be problematic. If we can stabilize the schema and stop changing them so much (which we really need to do anyway), this should not be so much of an issue.
> OR
> Be more liberal about adding new fields where folks need them, even if we are not supporting them in the UI right away. E.g., adding Language to a bunch of groups would cause us little pain, and then SMK could contribute templates that use the language fields.
> Neither of these are great, and I do not mean to be glib in suggesting them. Am trying to be practical and realistic about our current resource situation.
>
> HTH - Patrick
>
>
> From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
> Sent: Friday, April 20, 2012 9:01 AM
> To: Carly Bogen
> Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM; Patrick Schmitz
> Subject: Re: Concerns about 2.4
>
> Thanks, Carly. Yes, you're right. Patrick has been calling this work "merging" because the app layer needs to merge the extension fields into the core group somehow. This would be needed for PT/NPT if the common set of term fields were all in one shared schema. However, Patrick has acknowledged that the "common" fields could be in their own authority-specific schema. We just have extra work to try to make sure those fields don't diverge in the future. I might not be translating this correctly :-)
>
> Chris
>
> On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
>
>> Hi Chris,
>>
>> Based on the design and scope discussions we had this week, preferred and non-preferred terms and support for non-default vocabularies are the main drivers for 2.4. I'm not sure where the concern about extending core field groups came from, since it was not something we discussed. Is this related to the merging issue? If so, it was my understanding that fixing the merging issue was key to making PT and NPT and non-default vocabularies work.
>>
>> Patrick, maybe you can weigh in on this and help me understand.
>>
>> Thanks,
>> Carly
>>
>> On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman <chris.hoffman@berkeley.edu> wrote:
>> Hi all,
>>
>> I'm really, really happy to see how much work the team has done to make 2.4 more realistic. However at UCB we still have some concerns that I want to bring up with you and other deployers.
>>
>> In talking with Patrick and Richard yesterday, I'm concerned that the work to enable deployers to extend core field groups is going to swamp other areas of work that for us are absolutely critical. PAHMA needs to launch with version 2.4, or they will have to extend TMS licensing (probably costing tens of thousands of dollars) and answer to campus auditors.
>>
>> We also want to add fields to existing groups, but in the meantime, we have created our own versions of those groups as needed. The work Patrick described to me is hard, risky, and has other implications for data loading and reporting (reporting will be very hard and the approach might have to change completely to point iReport at the App layer instead of at Postgres tables). Patrick is the only one who can do this work now, and he is concerned about being on the critical path for something like this. He is off for 3 weeks starting in early June.
>>
>> For PAHMA to launch with 2.4, they need the preferred/non-preferred term stuff, the support for multiple vocabularies within a single authority, and a fix to the dollar-sign import bug. There is also some significant customization work here that will need help from Patrick's team (e.g., supporting range search on object numbers, "show me objects with numbers 91-202 through 91-292").
>>
>> I suspect field group extension is critical for SMK in particular. However, we really are on the edge of a major situation here.
>>
>> Thanks,
>> Chris
>>
>
PS
Patrick Schmitz
Mon, Apr 23, 2012 4:05 PM
Some quick answers before I head off to class:
What do we lose by not doing the merging work now?
The way I would describe it is: We lose one approach to doing the work you
describe. The approach we lose is simpler in services configuration, a net
wash in app layer and UI configuration, but more complex for other work like
reporting, export, batch processing, etc. It is also somewhat riskier to
maintain in the repository.
In addition, the work we are doing to put language into each term for
authorities, will be done regardless of approach. This work addresses a
major weakness for L14N of CSpace.
What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we
do not do the merging work now?
There is nothing I can think of that depends completely on this merging
work. It was originally conceived of as a way to simplify
customization/L14N, but we have identified flaws in that argument (the
issues with reporting, batch, etc.). We would have used this functionality
to implement the new PT/NPT support, but we do not have to. I know of
nothing else that must take this approach.
Two additional points:
-
2.4 will take more than two weeks in any case.
-
We should seriously consider the suggestion to add some additional
fields into our common schemas to mitigate the problems ChrisP et al. are
facing. I am not saying we do this quickly without discussion, as any
changes to the schemas need discussion. However, we can allow many changes
to the schemas (like adding language markers, or additional notes fields)
without requiring that the demo UI we ship expose all those fields. If this
makes sense, it might be a useful compromise.
HTH - Patrick
From: Angela Spinazze [mailto:ats@atspin.com]
Sent: Monday, April 23, 2012 7:34 AM
To: Patrick Schmitz; Chris Hoffman; Carly Bogen; Heather Hart
Cc: Carl Goodman; David A. Greenbaum; CollectionSpace Talk List
Subject: Re: Concerns about 2.4
Hi all and thanks for the discussion about 2.4. I have cc'd the Talk list
so that everyone can see this discussion as it is precisely the type of
conversation that we want to have happen in our community space.
What I take away from the discussion thread is that there is a tension
between taking on the merging work now versus shifting that work to a later
point in our development schedule so that the immediate needs of deployments
at UCB might instead be more efficiently addressed (based on current
resource constraints). The merging work has been talked about and pushed out
for over a year now so the landscape looks different from the deployer
perspective than it once did, yet the change to the model that the merging
work represents remains critical to the long-term viability of the software
for future deployers.
A couple of questions that come to mind are:
What do we lose by not doing the merging work now?
I think that the answer is (based on Patrick's comments below) that we lose
the ability to extend repeatable groups, and we lose support for
localization (neither of which we have now). Is that correct? Chris, you
pointed out that UCB has already addressed repeatable groups through a
work-around. Chris Pott has confirmed that he has had to do the same thing
at SMK and I suspect that MMI and Walker have had to as well (but I don't
know for certain- Jesse, Chris, Nate please chime in if you are so moved).
This means that while the merging work will make extending repeatable groups
easier, more efficient and effective in the future, current deployers won't
necessarily benefit from the change to the model right now. Same goes for
localization.
The other work defined for 2.4 ensures that we gain support for preferred
and non-preferred terms and support for non-default vocabularies and is
defined as a low-risk, achievable set of deliverables for a 2-week
development sprint.
What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we
do not do the merging work now?
I'm not sure that I know what the answer is to this question. Would we, for
example, still be able to work on relationships between catalog records (AKA
multiple object handling) in 2.7 without the merging work? Is anything
related to media handling(2.6) or improving the upgrade process (2.6)
affected by not doing the merging work?
If what we gain from pushing the merging work out to a later time in the
development schedule is more beneficial for all current deployers, then I
support the shift. It sounds like that is the case, but I need your help
Patrick with answers to the questions above, in order to fully understand
the ramifications.
Angela
On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
Yes, this is all correct so far.
We could take a messier approach with the PT/NPT changes. That would work,
but is a bit messier in a number of places. However, the messier approach is
well understood and low risk.
We had been planning to take an approach that was cleaner, but required
support for merging.
Yesterday, discussions about how we would actually implement the merging led
to some questions about how this would look in the database, and the
implications for report authoring. There was some surprise from our
deployers at how complex the so-called "merged" model would be in the DB.
For SMK, it will mean that any report authoring will be more complex if they
are injecting language all over.
Richard and I are feeling ambivalent about this. As software engineers, we
feel that the functionality should be there to support merging, and that it
would make the code for services and app cleaner, and would present a nice
model to the UI. However, being practical, and considering the additional
pain that this will entail for reporting, I am having some doubts. Knowing
how long it is taking me to get code working in the app layer, I am
concerned about the risk.
The details:
Let's say we want to add language to some repeating group of fields (let's
call it FooInfo). This will entail setting up specific fields for language
in an extension schema (it actually has to be something like
smk_FooInfoList/smk_FooInfo/language), some associated config in the app
layer, and the obvious UI work to present it to the user.
What happens on the back-end is that the new language fields end up in a
separate table (smk_FooInfo) that the app layer and services have to keep
aligned with the FooInfo part of the common schema. The UI and app layer
should be able to ensure that if there are three rows in FooInfo, there will
be three rows in smk_FooInfo. This is the code I would have to write, and it
entails changes to the core of the app layer, so is very high risk.
Issues:
If someone uses the REST APIs or import and does not get this right,
everything will get hosed (e.g., SMK's language markers will end up on the
wrong row for the notes they were supposed to be associated with. There is
no reasonable way for services to even detect when an import or REST call
has screwed this up.
*
When someone wants to generate a report that includes the language
(or selects for the language) for a note, the query will get a little funky,
requiring more careful joins and constraints in what is already a complex
data model.
ChrisH was surprised that the fields would not be merged into the common
table, until we explained that it cannot without changing the common model
in the code and for all the users. Even if we split up the repository to be
one per tenant, our code (a lot of the the REST stuff) would get confused by
not having a consistent definition of the schema, and so we have to add a
new table for every injected field.
One alternative is to defer or just not support merging. Instead, we would
either:
Tell folks to live with the model they have now, in which they have
to redefine the whole repeating structure everywhere they want to add a
field. Ugly, but well understood. The main objection to this after the
amount of typing, was that upgrades could be problematic. If we can
stabilize the schema and stop changing them so much (which we really need to
do anyway), this should not be so much of an issue.
OR
2.
Be more liberal about adding new fields where folks need them, even
if we are not supporting them in the UI right away. E.g., adding Language to
a bunch of groups would cause us little pain, and then SMK could contribute
templates that use the language fields.
Neither of these are great, and I do not mean to be glib in suggesting them.
Am trying to be practical and realistic about our current resource
situation.
HTH - Patrick
From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
Sent: Friday, April 20, 2012 9:01 AM
To: Carly Bogen
Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM; Patrick
Schmitz
Subject: Re: Concerns about 2.4
Thanks, Carly. Yes, you're right. Patrick has been calling this work
"merging" because the app layer needs to merge the extension fields into the
core group somehow. This would be needed for PT/NPT if the common set of
term fields were all in one shared schema. However, Patrick has
acknowledged that the "common" fields could be in their own
authority-specific schema. We just have extra work to try to make sure
those fields don't diverge in the future. I might not be translating this
correctly :-)
Chris
On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
Hi Chris,
Based on the design and scope discussions we had this week, preferred and
non-preferred terms and support for non-default vocabularies are the main
drivers for 2.4. I'm not sure where the concern about extending core field
groups came from, since it was not something we discussed. Is this related
to the merging issue? If so, it was my understanding that fixing the
merging issue was key to making PT and NPT and non-default vocabularies
work.
Patrick, maybe you can weigh in on this and help me understand.
Thanks,
Carly
On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman chris.hoffman@berkeley.edu
wrote:
Hi all,
I'm really, really happy to see how much work the team has done to make 2.4
more realistic. However at UCB we still have some concerns that I want to
bring up with you and other deployers.
In talking with Patrick and Richard yesterday, I'm concerned that the work
to enable deployers to extend core field groups is going to swamp other
areas of work that for us are absolutely critical. PAHMA needs to launch
with version 2.4, or they will have to extend TMS licensing (probably
costing tens of thousands of dollars) and answer to campus auditors.
We also want to add fields to existing groups, but in the meantime, we have
created our own versions of those groups as needed. The work Patrick
described to me is hard, risky, and has other implications for data loading
and reporting (reporting will be very hard and the approach might have to
change completely to point iReport at the App layer instead of at Postgres
tables). Patrick is the only one who can do this work now, and he is
concerned about being on the critical path for something like this. He is
off for 3 weeks starting in early June.
For PAHMA to launch with 2.4, they need the preferred/non-preferred term
stuff, the support for multiple vocabularies within a single authority, and
a fix to the dollar-sign import bug. There is also some significant
customization work here that will need help from Patrick's team (e.g.,
supporting range search on object numbers, "show me objects with numbers
91-202 through 91-292").
I suspect field group extension is critical for SMK in particular. However,
we really are on the edge of a major situation here.
Thanks,
Chris
Some quick answers before I head off to class:
What do we lose by not doing the merging work now?
The way I would describe it is: We lose one approach to doing the work you
describe. The approach we lose is simpler in services configuration, a net
wash in app layer and UI configuration, but more complex for other work like
reporting, export, batch processing, etc. It is also somewhat riskier to
maintain in the repository.
In addition, the work we are doing to put language into each term for
authorities, will be done regardless of approach. This work addresses a
major weakness for L14N of CSpace.
What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we
do not do the merging work now?
There is nothing I can think of that depends completely on this merging
work. It was originally conceived of as a way to simplify
customization/L14N, but we have identified flaws in that argument (the
issues with reporting, batch, etc.). We would have used this functionality
to implement the new PT/NPT support, but we do not have to. I know of
nothing else that must take this approach.
Two additional points:
1. 2.4 will take more than two weeks in any case.
2. We should seriously consider the suggestion to add some additional
fields into our common schemas to mitigate the problems ChrisP et al. are
facing. I am not saying we do this quickly without discussion, as any
changes to the schemas need discussion. However, we can allow many changes
to the schemas (like adding language markers, or additional notes fields)
without requiring that the demo UI we ship expose all those fields. If this
makes sense, it might be a useful compromise.
HTH - Patrick
_____
From: Angela Spinazze [mailto:ats@atspin.com]
Sent: Monday, April 23, 2012 7:34 AM
To: Patrick Schmitz; Chris Hoffman; Carly Bogen; Heather Hart
Cc: Carl Goodman; David A. Greenbaum; CollectionSpace Talk List
Subject: Re: Concerns about 2.4
Hi all and thanks for the discussion about 2.4. I have cc'd the Talk list
so that everyone can see this discussion as it is precisely the type of
conversation that we want to have happen in our community space.
What I take away from the discussion thread is that there is a tension
between taking on the merging work now versus shifting that work to a later
point in our development schedule so that the immediate needs of deployments
at UCB might instead be more efficiently addressed (based on current
resource constraints). The merging work has been talked about and pushed out
for over a year now so the landscape looks different from the deployer
perspective than it once did, yet the change to the model that the merging
work represents remains critical to the long-term viability of the software
for future deployers.
A couple of questions that come to mind are:
What do we lose by not doing the merging work now?
I think that the answer is (based on Patrick's comments below) that we lose
the ability to extend repeatable groups, and we lose support for
localization (neither of which we have now). Is that correct? Chris, you
pointed out that UCB has already addressed repeatable groups through a
work-around. Chris Pott has confirmed that he has had to do the same thing
at SMK and I suspect that MMI and Walker have had to as well (but I don't
know for certain- Jesse, Chris, Nate please chime in if you are so moved).
This means that while the merging work will make extending repeatable groups
easier, more efficient and effective in the future, current deployers won't
necessarily benefit from the change to the model right now. Same goes for
localization.
The other work defined for 2.4 ensures that we gain support for preferred
and non-preferred terms and support for non-default vocabularies and is
defined as a low-risk, achievable set of deliverables for a 2-week
development sprint.
What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we
do not do the merging work now?
I'm not sure that I know what the answer is to this question. Would we, for
example, still be able to work on relationships between catalog records (AKA
multiple object handling) in 2.7 without the merging work? Is anything
related to media handling(2.6) or improving the upgrade process (2.6)
affected by not doing the merging work?
If what we gain from pushing the merging work out to a later time in the
development schedule is more beneficial for all current deployers, then I
support the shift. It sounds like that is the case, but I need your help
Patrick with answers to the questions above, in order to fully understand
the ramifications.
Angela
On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
Yes, this is all correct so far.
We *could* take a messier approach with the PT/NPT changes. That would work,
but is a bit messier in a number of places. However, the messier approach is
well understood and low risk.
We had been planning to take an approach that was cleaner, but required
support for merging.
Yesterday, discussions about how we would actually implement the merging led
to some questions about how this would look in the database, and the
implications for report authoring. There was some surprise from our
deployers at how complex the so-called "merged" model would be in the DB.
For SMK, it will mean that any report authoring will be more complex if they
are injecting language all over.
Richard and I are feeling ambivalent about this. As software engineers, we
feel that the functionality should be there to support merging, and that it
would make the code for services and app cleaner, and would present a nice
model to the UI. However, being practical, and considering the additional
pain that this will entail for reporting, I am having some doubts. Knowing
how long it is taking me to get code working in the app layer, I am
concerned about the risk.
The details:
Let's say we want to add language to some repeating group of fields (let's
call it FooInfo). This will entail setting up specific fields for language
in an extension schema (it actually has to be something like
smk_FooInfoList/smk_FooInfo/language), some associated config in the app
layer, and the obvious UI work to present it to the user.
What happens on the back-end is that the new language fields end up in a
separate table (smk_FooInfo) that the app layer and services have to keep
aligned with the FooInfo part of the common schema. The UI and app layer
should be able to ensure that if there are three rows in FooInfo, there will
be three rows in smk_FooInfo. This is the code I would have to write, and it
entails changes to the core of the app layer, so is very high risk.
Issues:
*
If someone uses the REST APIs or import and does not get this right,
everything will get hosed (e.g., SMK's language markers will end up on the
wrong row for the notes they were supposed to be associated with. There is
no reasonable way for services to even detect when an import or REST call
has screwed this up.
*
When someone wants to generate a report that includes the language
(or selects for the language) for a note, the query will get a little funky,
requiring more careful joins and constraints in what is already a complex
data model.
ChrisH was surprised that the fields would not be merged into the common
table, until we explained that it cannot without changing the common model
in the code and for all the users. Even if we split up the repository to be
one per tenant, our code (a lot of the the REST stuff) would get confused by
not having a consistent definition of the schema, and so we have to add a
new table for every injected field.
One alternative is to defer or just not support merging. Instead, we would
either:
1.
Tell folks to live with the model they have now, in which they have
to redefine the whole repeating structure everywhere they want to add a
field. Ugly, but well understood. The main objection to this after the
amount of typing, was that upgrades could be problematic. If we can
stabilize the schema and stop changing them so much (which we really need to
do anyway), this should not be so much of an issue.
OR
2.
Be more liberal about adding new fields where folks need them, even
if we are not supporting them in the UI right away. E.g., adding Language to
a bunch of groups would cause us little pain, and then SMK could contribute
templates that use the language fields.
Neither of these are great, and I do not mean to be glib in suggesting them.
Am trying to be practical and realistic about our current resource
situation.
HTH - Patrick
_____
From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
Sent: Friday, April 20, 2012 9:01 AM
To: Carly Bogen
Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM; Patrick
Schmitz
Subject: Re: Concerns about 2.4
Thanks, Carly. Yes, you're right. Patrick has been calling this work
"merging" because the app layer needs to merge the extension fields into the
core group somehow. This would be needed for PT/NPT if the common set of
term fields were all in one shared schema. However, Patrick has
acknowledged that the "common" fields could be in their own
authority-specific schema. We just have extra work to try to make sure
those fields don't diverge in the future. I might not be translating this
correctly :-)
Chris
On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
Hi Chris,
Based on the design and scope discussions we had this week, preferred and
non-preferred terms and support for non-default vocabularies are the main
drivers for 2.4. I'm not sure where the concern about extending core field
groups came from, since it was not something we discussed. Is this related
to the merging issue? If so, it was my understanding that fixing the
merging issue was key to making PT and NPT and non-default vocabularies
work.
Patrick, maybe you can weigh in on this and help me understand.
Thanks,
Carly
On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman <chris.hoffman@berkeley.edu>
wrote:
Hi all,
I'm really, really happy to see how much work the team has done to make 2.4
more realistic. However at UCB we still have some concerns that I want to
bring up with you and other deployers.
In talking with Patrick and Richard yesterday, I'm concerned that the work
to enable deployers to extend core field groups is going to swamp other
areas of work that for us are absolutely critical. PAHMA needs to launch
with version 2.4, or they will have to extend TMS licensing (probably
costing tens of thousands of dollars) and answer to campus auditors.
We also want to add fields to existing groups, but in the meantime, we have
created our own versions of those groups as needed. The work Patrick
described to me is hard, risky, and has other implications for data loading
and reporting (reporting will be very hard and the approach might have to
change completely to point iReport at the App layer instead of at Postgres
tables). Patrick is the only one who can do this work now, and he is
concerned about being on the critical path for something like this. He is
off for 3 weeks starting in early June.
For PAHMA to launch with 2.4, they need the preferred/non-preferred term
stuff, the support for multiple vocabularies within a single authority, and
a fix to the dollar-sign import bug. There is also some significant
customization work here that will need help from Patrick's team (e.g.,
supporting range search on object numbers, "show me objects with numbers
91-202 through 91-292").
I suspect field group extension is critical for SMK in particular. However,
we really are on the edge of a major situation here.
Thanks,
Chris
JM
Jesse Martinez
Mon, Apr 23, 2012 10:23 PM
Here at MMI, we are also using a work-around for extending repeatable
groups in the schema. I'm fine with continuing the current work-around
model and having a known yet stable db structure even if the process
is "messy"; we only have a few instances of this work-around in use so
it's manageable from our perspective. (For other institutions this may
not be a luxury.) Like Patrick mentioned, the biggest concerns I have
with the current model are upgrades and schema changes to these
repeatable fields, but this is a minimal problem due to the dearth of
work-arounds we have in use. It would be nice to have the merging
model in use, but it's not critical for us at the present.
On Mon, Apr 23, 2012 at 10:33 AM, Angela Spinazze ats@atspin.com wrote:
Hi all and thanks for the discussion about 2.4. I have cc'd the Talk list
so that everyone can see this discussion as it is precisely the type of
conversation that we want to have happen in our community space.
What I take away from the discussion thread is that there is a tension
between taking on the merging work now versus shifting that work to a later
point in our development schedule so that the immediate needs of deployments
at UCB might instead be more efficiently addressed (based on current
resource constraints). The merging work has been talked about and pushed out
for over a year now so the landscape looks different from the deployer
perspective than it once did, yet the change to the model that the merging
work represents remains critical to the long-term viability of the software
for future deployers.
A couple of questions that come to mind are:
What do we lose by not doing the merging work now?
I think that the answer is (based on Patrick's comments below) that we lose
the ability to extend repeatable groups, and we lose support for
localization (neither of which we have now). Is that correct? Chris, you
pointed out that UCB has already addressed repeatable groups through a
work-around. Chris Pott has confirmed that he has had to do the same thing
at SMK and I suspect that MMI and Walker have had to as well (but I don't
know for certain- Jesse, Chris, Nate please chime in if you are so moved).
This means that while the merging work will make extending repeatable
groups easier, more efficient and effective in the future, current deployers
won't necessarily benefit from the change to the model right now. Same goes
for localization.
The other work defined for 2.4 ensures that we gain support for preferred
and non-preferred terms and support for non-default vocabularies and is
defined as a low-risk, achievable set of deliverables for a 2-week
development sprint.
What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we
do not do the merging work now?
I'm not sure that I know what the answer is to this question. Would we, for
example, still be able to work on relationships between catalog records (AKA
multiple object handling) in 2.7 without the merging work? Is anything
related to media handling(2.6) or improving the upgrade process (2.6)
affected by not doing the merging work?
If what we gain from pushing the merging work out to a later time in the
development schedule is more beneficial for all current deployers, then I
support the shift. It sounds like that is the case, but I need your help
Patrick with answers to the questions above, in order to fully understand
the ramifications.
Angela
On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
Yes, this is all correct so far.
We could take a messier approach with the PT/NPT changes. That would work,
but is a bit messier in a number of places. However, the messier approach is
well understood and low risk.
We had been planning to take an approach that was cleaner, but required
support for merging.
Yesterday, discussions about how we would actually implement the merging led
to some questions about how this would look in the database, and the
implications for report authoring. There was some surprise from our
deployers at how complex the so-called "merged" model would be in the DB.
For SMK, it will mean that any report authoring will be more complex if they
are injecting language all over.
Richard and I are feeling ambivalent about this. As software engineers, we
feel that the functionality should be there to support merging, and that it
would make the code for services and app cleaner, and would present a nice
model to the UI. However, being practical, and considering the additional
pain that this will entail for reporting, I am having some doubts. Knowing
how long it is taking me to get code working in the app layer, I am
concerned about the risk.
The details:
Let's say we want to add language to some repeating group of fields (let's
call it FooInfo). This will entail setting up specific fields for language
in an extension schema (it actually has to be something like
smk_FooInfoList/smk_FooInfo/language), some associated config in the app
layer, and the obvious UI work to present it to the user.
What happens on the back-end is that the new language fields end up in a
separate table (smk_FooInfo) that the app layer and services have to keep
aligned with the FooInfo part of the common schema. The UI and app layer
should be able to ensure that if there are three rows in FooInfo, there will
be three rows in smk_FooInfo. This is the code I would have to write, and
it entails changes to the core of the app layer, so is very high risk.
Issues:
If someone uses the REST APIs or import and does not get this right,
everything will get hosed (e.g., SMK's language markers will end up on the
wrong row for the notes they were supposed to be associated with. There is
no reasonable way for services to even detect when an import or REST call
has screwed this up.
When someone wants to generate a report that includes the language (or
selects for the language) for a note, the query will get a little funky,
requiring more careful joins and constraints in what is already a complex
data model.
ChrisH was surprised that the fields would not be merged into the common
table, until we explained that it cannot without changing the common model
in the code and for all the users. Even if we split up the repository to be
one per tenant, our code (a lot of the the REST stuff) would get confused by
not having a consistent definition of the schema, and so we have to add a
new table for every injected field.
One alternative is to defer or just not support merging. Instead, we would
either:
Tell folks to live with the model they have now, in which they have to
redefine the whole repeating structure everywhere they want to add a field.
Ugly, but well understood. The main objection to this after the amount of
typing, was that upgrades could be problematic. If we can stabilize the
schema and stop changing them so much (which we really need to do anyway),
this should not be so much of an issue.
OR
Be more liberal about adding new fields where folks need them, even if we
are not supporting them in the UI right away. E.g., adding Language to a
bunch of groups would cause us little pain, and then SMK could contribute
templates that use the language fields.
Neither of these are great, and I do not mean to be glib in suggesting them.
Am trying to be practical and realistic about our current resource
situation.
HTH - Patrick
From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
Sent: Friday, April 20, 2012 9:01 AM
To: Carly Bogen
Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM; Patrick
Schmitz
Subject: Re: Concerns about 2.4
Thanks, Carly. Yes, you're right. Patrick has been calling this work
"merging" because the app layer needs to merge the extension fields into the
core group somehow. This would be needed for PT/NPT if the common set of
term fields were all in one shared schema. However, Patrick has
acknowledged that the "common" fields could be in their own
authority-specific schema. We just have extra work to try to make sure
those fields don't diverge in the future. I might not be translating this
correctly :-)
Chris
On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
Hi Chris,
Based on the design and scope discussions we had this week, preferred and
non-preferred terms and support for non-default vocabularies are the main
drivers for 2.4. I'm not sure where the concern about extending core field
groups came from, since it was not something we discussed. Is this related
to the merging issue? If so, it was my understanding that fixing the
merging issue was key to making PT and NPT and non-default vocabularies
work.
Patrick, maybe you can weigh in on this and help me understand.
Thanks,
Carly
On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman chris.hoffman@berkeley.edu
wrote:
Hi all,
I'm really, really happy to see how much work the team has done to make
2.4 more realistic. However at UCB we still have some concerns that I want
to bring up with you and other deployers.
In talking with Patrick and Richard yesterday, I'm concerned that the work
to enable deployers to extend core field groups is going to swamp other
areas of work that for us are absolutely critical. PAHMA needs to launch
with version 2.4, or they will have to extend TMS licensing (probably
costing tens of thousands of dollars) and answer to campus auditors.
We also want to add fields to existing groups, but in the meantime, we
have created our own versions of those groups as needed. The work Patrick
described to me is hard, risky, and has other implications for data loading
and reporting (reporting will be very hard and the approach might have to
change completely to point iReport at the App layer instead of at Postgres
tables). Patrick is the only one who can do this work now, and he is
concerned about being on the critical path for something like this. He is
off for 3 weeks starting in early June.
For PAHMA to launch with 2.4, they need the preferred/non-preferred term
stuff, the support for multiple vocabularies within a single authority, and
a fix to the dollar-sign import bug. There is also some significant
customization work here that will need help from Patrick's team (e.g.,
supporting range search on object numbers, "show me objects with numbers
91-202 through 91-292").
I suspect field group extension is critical for SMK in particular.
However, we really are on the edge of a major situation here.
Thanks,
Chris
Here at MMI, we are also using a work-around for extending repeatable
groups in the schema. I'm fine with continuing the current work-around
model and having a known yet stable db structure even if the process
is "messy"; we only have a few instances of this work-around in use so
it's manageable from our perspective. (For other institutions this may
not be a luxury.) Like Patrick mentioned, the biggest concerns I have
with the current model are upgrades and schema changes to these
repeatable fields, but this is a minimal problem due to the dearth of
work-arounds we have in use. It would be nice to have the merging
model in use, but it's not critical for us at the present.
- Jesse
On Mon, Apr 23, 2012 at 10:33 AM, Angela Spinazze <ats@atspin.com> wrote:
> Hi all and thanks for the discussion about 2.4. I have cc'd the Talk list
> so that everyone can see this discussion as it is precisely the type of
> conversation that we want to have happen in our community space.
>
> What I take away from the discussion thread is that there is a tension
> between taking on the merging work now versus shifting that work to a later
> point in our development schedule so that the immediate needs of deployments
> at UCB might instead be more efficiently addressed (based on current
> resource constraints). The merging work has been talked about and pushed out
> for over a year now so the landscape looks different from the deployer
> perspective than it once did, yet the change to the model that the merging
> work represents remains critical to the long-term viability of the software
> for future deployers.
>
> A couple of questions that come to mind are:
>
> What do we lose by not doing the merging work now?
> I think that the answer is (based on Patrick's comments below) that we lose
> the ability to extend repeatable groups, and we lose support for
> localization (neither of which we have now). Is that correct? Chris, you
> pointed out that UCB has already addressed repeatable groups through a
> work-around. Chris Pott has confirmed that he has had to do the same thing
> at SMK and I suspect that MMI and Walker have had to as well (but I don't
> know for certain- Jesse, Chris, Nate please chime in if you are so moved).
> This means that while the merging work will make extending repeatable
> groups easier, more efficient and effective in the future, current deployers
> won't necessarily benefit from the change to the model right now. Same goes
> for localization.
>
> The other work defined for 2.4 ensures that we gain support for preferred
> and non-preferred terms and support for non-default vocabularies and is
> defined as a low-risk, achievable set of deliverables for a 2-week
> development sprint.
>
> What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we
> do not do the merging work now?
> I'm not sure that I know what the answer is to this question. Would we, for
> example, still be able to work on relationships between catalog records (AKA
> multiple object handling) in 2.7 without the merging work? Is anything
> related to media handling(2.6) or improving the upgrade process (2.6)
> affected by not doing the merging work?
>
> If what we gain from pushing the merging work out to a later time in the
> development schedule is more beneficial for all current deployers, then I
> support the shift. It sounds like that is the case, but I need your help
> Patrick with answers to the questions above, in order to fully understand
> the ramifications.
>
> Angela
>
> On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
>
> Yes, this is all correct so far.
>
> We *could* take a messier approach with the PT/NPT changes. That would work,
> but is a bit messier in a number of places. However, the messier approach is
> well understood and low risk.
>
> We had been planning to take an approach that was cleaner, but required
> support for merging.
>
> Yesterday, discussions about how we would actually implement the merging led
> to some questions about how this would look in the database, and the
> implications for report authoring. There was some surprise from our
> deployers at how complex the so-called "merged" model would be in the DB.
>
> For SMK, it will mean that any report authoring will be more complex if they
> are injecting language all over.
>
> Richard and I are feeling ambivalent about this. As software engineers, we
> feel that the functionality should be there to support merging, and that it
> would make the code for services and app cleaner, and would present a nice
> model to the UI. However, being practical, and considering the additional
> pain that this will entail for reporting, I am having some doubts. Knowing
> how long it is taking me to get code working in the app layer, I am
> concerned about the risk.
>
> The details:
>
> Let's say we want to add language to some repeating group of fields (let's
> call it FooInfo). This will entail setting up specific fields for language
> in an extension schema (it actually has to be something like
> smk_FooInfoList/smk_FooInfo/language), some associated config in the app
> layer, and the obvious UI work to present it to the user.
>
> What happens on the back-end is that the new language fields end up in a
> separate table (smk_FooInfo) that the app layer and services have to keep
> aligned with the FooInfo part of the common schema. The UI and app layer
> should be able to ensure that if there are three rows in FooInfo, there will
> be three rows in smk_FooInfo. This is the code I would have to write, and
> it entails changes to the core of the app layer, so is very high risk.
>
> Issues:
>
> If someone uses the REST APIs or import and does not get this right,
> everything will get hosed (e.g., SMK's language markers will end up on the
> wrong row for the notes they were supposed to be associated with. There is
> no reasonable way for services to even detect when an import or REST call
> has screwed this up.
> When someone wants to generate a report that includes the language (or
> selects for the language) for a note, the query will get a little funky,
> requiring more careful joins and constraints in what is already a complex
> data model.
>
> ChrisH was surprised that the fields would not be merged into the common
> table, until we explained that it cannot without changing the common model
> in the code and for all the users. Even if we split up the repository to be
> one per tenant, our code (a lot of the the REST stuff) would get confused by
> not having a consistent definition of the schema, and so we have to add a
> new table for every injected field.
>
> One alternative is to defer or just not support merging. Instead, we would
> either:
>
> Tell folks to live with the model they have now, in which they have to
> redefine the whole repeating structure everywhere they want to add a field.
> Ugly, but well understood. The main objection to this after the amount of
> typing, was that upgrades could be problematic. If we can stabilize the
> schema and stop changing them so much (which we really need to do anyway),
> this should not be so much of an issue.
> OR
> Be more liberal about adding new fields where folks need them, even if we
> are not supporting them in the UI right away. E.g., adding Language to a
> bunch of groups would cause us little pain, and then SMK could contribute
> templates that use the language fields.
>
> Neither of these are great, and I do not mean to be glib in suggesting them.
> Am trying to be practical and realistic about our current resource
> situation.
>
> HTH - Patrick
>
>
> ________________________________
> From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
> Sent: Friday, April 20, 2012 9:01 AM
> To: Carly Bogen
> Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM; Patrick
> Schmitz
> Subject: Re: Concerns about 2.4
>
> Thanks, Carly. Yes, you're right. Patrick has been calling this work
> "merging" because the app layer needs to merge the extension fields into the
> core group somehow. This would be needed for PT/NPT if the common set of
> term fields were all in one shared schema. However, Patrick has
> acknowledged that the "common" fields could be in their own
> authority-specific schema. We just have extra work to try to make sure
> those fields don't diverge in the future. I might not be translating this
> correctly :-)
>
> Chris
>
> On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
>
> Hi Chris,
>
> Based on the design and scope discussions we had this week, preferred and
> non-preferred terms and support for non-default vocabularies are the main
> drivers for 2.4. I'm not sure where the concern about extending core field
> groups came from, since it was not something we discussed. Is this related
> to the merging issue? If so, it was my understanding that fixing the
> merging issue was key to making PT and NPT and non-default vocabularies
> work.
>
> Patrick, maybe you can weigh in on this and help me understand.
>
> Thanks,
> Carly
>
> On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman <chris.hoffman@berkeley.edu>
> wrote:
>>
>> Hi all,
>>
>> I'm really, really happy to see how much work the team has done to make
>> 2.4 more realistic. However at UCB we still have some concerns that I want
>> to bring up with you and other deployers.
>>
>> In talking with Patrick and Richard yesterday, I'm concerned that the work
>> to enable deployers to extend core field groups is going to swamp other
>> areas of work that for us are absolutely critical. PAHMA needs to launch
>> with version 2.4, or they will have to extend TMS licensing (probably
>> costing tens of thousands of dollars) and answer to campus auditors.
>>
>> We also want to add fields to existing groups, but in the meantime, we
>> have created our own versions of those groups as needed. The work Patrick
>> described to me is hard, risky, and has other implications for data loading
>> and reporting (reporting will be very hard and the approach might have to
>> change completely to point iReport at the App layer instead of at Postgres
>> tables). Patrick is the only one who can do this work now, and he is
>> concerned about being on the critical path for something like this. He is
>> off for 3 weeks starting in early June.
>>
>> For PAHMA to launch with 2.4, they need the preferred/non-preferred term
>> stuff, the support for multiple vocabularies within a single authority, and
>> a fix to the dollar-sign import bug. There is also some significant
>> customization work here that will need help from Patrick's team (e.g.,
>> supporting range search on object numbers, "show me objects with numbers
>> 91-202 through 91-292").
>>
>> I suspect field group extension is critical for SMK in particular.
>> However, we really are on the edge of a major situation here.
>>
>> Thanks,
>> Chris
>
>
>
>
>
> _______________________________________________
> Talk mailing list
> Talk@lists.collectionspace.org
> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org
>
CG
Carl Goodman
Tue, Apr 24, 2012 2:02 PM
Please answer this for me: what are the implications on an org/co that is
HOSTING the software for others? And will those others be adversely
impacted?
Thanks,
Carl
On Mon, Apr 23, 2012 at 6:23 PM, Jesse Martinez jmartinez@movingimage.uswrote:
Here at MMI, we are also using a work-around for extending repeatable
groups in the schema. I'm fine with continuing the current work-around
model and having a known yet stable db structure even if the process
is "messy"; we only have a few instances of this work-around in use so
it's manageable from our perspective. (For other institutions this may
not be a luxury.) Like Patrick mentioned, the biggest concerns I have
with the current model are upgrades and schema changes to these
repeatable fields, but this is a minimal problem due to the dearth of
work-arounds we have in use. It would be nice to have the merging
model in use, but it's not critical for us at the present.
On Mon, Apr 23, 2012 at 10:33 AM, Angela Spinazze ats@atspin.com wrote:
Hi all and thanks for the discussion about 2.4. I have cc'd the Talk
so that everyone can see this discussion as it is precisely the type of
conversation that we want to have happen in our community space.
What I take away from the discussion thread is that there is a tension
between taking on the merging work now versus shifting that work to a
point in our development schedule so that the immediate needs of
at UCB might instead be more efficiently addressed (based on current
resource constraints). The merging work has been talked about and pushed
for over a year now so the landscape looks different from the deployer
perspective than it once did, yet the change to the model that the
work represents remains critical to the long-term viability of the
for future deployers.
A couple of questions that come to mind are:
What do we lose by not doing the merging work now?
I think that the answer is (based on Patrick's comments below) that we
the ability to extend repeatable groups, and we lose support for
localization (neither of which we have now). Is that correct? Chris,
pointed out that UCB has already addressed repeatable groups through a
work-around. Chris Pott has confirmed that he has had to do the same
at SMK and I suspect that MMI and Walker have had to as well (but I don't
know for certain- Jesse, Chris, Nate please chime in if you are so
This means that while the merging work will make extending repeatable
groups easier, more efficient and effective in the future, current
won't necessarily benefit from the change to the model right now. Same
for localization.
The other work defined for 2.4 ensures that we gain support for preferred
and non-preferred terms and support for non-default vocabularies and is
defined as a low-risk, achievable set of deliverables for a 2-week
development sprint.
What is prevented from being developed in releases 2.6, 2.7, and 2.8 if
do not do the merging work now?
I'm not sure that I know what the answer is to this question. Would we,
example, still be able to work on relationships between catalog records
multiple object handling) in 2.7 without the merging work? Is anything
related to media handling(2.6) or improving the upgrade process (2.6)
affected by not doing the merging work?
If what we gain from pushing the merging work out to a later time in the
development schedule is more beneficial for all current deployers, then I
support the shift. It sounds like that is the case, but I need your help
Patrick with answers to the questions above, in order to fully understand
the ramifications.
Angela
On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
Yes, this is all correct so far.
We could take a messier approach with the PT/NPT changes. That would
but is a bit messier in a number of places. However, the messier
well understood and low risk.
We had been planning to take an approach that was cleaner, but required
support for merging.
Yesterday, discussions about how we would actually implement the merging
to some questions about how this would look in the database, and the
implications for report authoring. There was some surprise from our
deployers at how complex the so-called "merged" model would be in the DB.
For SMK, it will mean that any report authoring will be more complex if
are injecting language all over.
Richard and I are feeling ambivalent about this. As software engineers,
feel that the functionality should be there to support merging, and that
would make the code for services and app cleaner, and would present a
model to the UI. However, being practical, and considering the additional
pain that this will entail for reporting, I am having some doubts.
how long it is taking me to get code working in the app layer, I am
concerned about the risk.
The details:
Let's say we want to add language to some repeating group of fields
call it FooInfo). This will entail setting up specific fields for
in an extension schema (it actually has to be something like
smk_FooInfoList/smk_FooInfo/language), some associated config in the app
layer, and the obvious UI work to present it to the user.
What happens on the back-end is that the new language fields end up in a
separate table (smk_FooInfo) that the app layer and services have to keep
aligned with the FooInfo part of the common schema. The UI and app layer
should be able to ensure that if there are three rows in FooInfo, there
be three rows in smk_FooInfo. This is the code I would have to write, and
it entails changes to the core of the app layer, so is very high risk.
Issues:
If someone uses the REST APIs or import and does not get this right,
everything will get hosed (e.g., SMK's language markers will end up on
wrong row for the notes they were supposed to be associated with. There
no reasonable way for services to even detect when an import or REST call
has screwed this up.
When someone wants to generate a report that includes the language (or
selects for the language) for a note, the query will get a little funky,
requiring more careful joins and constraints in what is already a complex
data model.
ChrisH was surprised that the fields would not be merged into the common
table, until we explained that it cannot without changing the common
in the code and for all the users. Even if we split up the repository to
one per tenant, our code (a lot of the the REST stuff) would get
not having a consistent definition of the schema, and so we have to add a
new table for every injected field.
One alternative is to defer or just not support merging. Instead, we
either:
Tell folks to live with the model they have now, in which they have to
redefine the whole repeating structure everywhere they want to add a
Ugly, but well understood. The main objection to this after the amount of
typing, was that upgrades could be problematic. If we can stabilize the
schema and stop changing them so much (which we really need to do
this should not be so much of an issue.
OR
Be more liberal about adding new fields where folks need them, even if we
are not supporting them in the UI right away. E.g., adding Language to a
bunch of groups would cause us little pain, and then SMK could contribute
templates that use the language fields.
Neither of these are great, and I do not mean to be glib in suggesting
Am trying to be practical and realistic about our current resource
situation.
HTH - Patrick
From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
Sent: Friday, April 20, 2012 9:01 AM
To: Carly Bogen
Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM;
Schmitz
Subject: Re: Concerns about 2.4
Thanks, Carly. Yes, you're right. Patrick has been calling this work
"merging" because the app layer needs to merge the extension fields into
core group somehow. This would be needed for PT/NPT if the common set of
term fields were all in one shared schema. However, Patrick has
acknowledged that the "common" fields could be in their own
authority-specific schema. We just have extra work to try to make sure
those fields don't diverge in the future. I might not be translating
correctly :-)
Chris
On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
Hi Chris,
Based on the design and scope discussions we had this week, preferred and
non-preferred terms and support for non-default vocabularies are the main
drivers for 2.4. I'm not sure where the concern about extending core
groups came from, since it was not something we discussed. Is this
to the merging issue? If so, it was my understanding that fixing the
merging issue was key to making PT and NPT and non-default vocabularies
work.
Patrick, maybe you can weigh in on this and help me understand.
Thanks,
Carly
On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman <
Hi all,
I'm really, really happy to see how much work the team has done to make
2.4 more realistic. However at UCB we still have some concerns that I
to bring up with you and other deployers.
In talking with Patrick and Richard yesterday, I'm concerned that the
to enable deployers to extend core field groups is going to swamp other
areas of work that for us are absolutely critical. PAHMA needs to
with version 2.4, or they will have to extend TMS licensing (probably
costing tens of thousands of dollars) and answer to campus auditors.
We also want to add fields to existing groups, but in the meantime, we
have created our own versions of those groups as needed. The work
described to me is hard, risky, and has other implications for data
and reporting (reporting will be very hard and the approach might have
change completely to point iReport at the App layer instead of at
tables). Patrick is the only one who can do this work now, and he is
concerned about being on the critical path for something like this. He
off for 3 weeks starting in early June.
For PAHMA to launch with 2.4, they need the preferred/non-preferred term
stuff, the support for multiple vocabularies within a single authority,
a fix to the dollar-sign import bug. There is also some significant
customization work here that will need help from Patrick's team (e.g.,
supporting range search on object numbers, "show me objects with numbers
91-202 through 91-292").
I suspect field group extension is critical for SMK in particular.
However, we really are on the edge of a major situation here.
Thanks,
Chris
--
Carl Goodman
Executive Director
MUSEUM OF THE MOVING IMAGE
36-01 35 Avenue, Astoria, NY 11106
movingimage.us 718 777 6800
Please answer this for me: what are the implications on an org/co that is
HOSTING the software for others? And will those others be adversely
impacted?
Thanks,
Carl
On Mon, Apr 23, 2012 at 6:23 PM, Jesse Martinez <jmartinez@movingimage.us>wrote:
> Here at MMI, we are also using a work-around for extending repeatable
> groups in the schema. I'm fine with continuing the current work-around
> model and having a known yet stable db structure even if the process
> is "messy"; we only have a few instances of this work-around in use so
> it's manageable from our perspective. (For other institutions this may
> not be a luxury.) Like Patrick mentioned, the biggest concerns I have
> with the current model are upgrades and schema changes to these
> repeatable fields, but this is a minimal problem due to the dearth of
> work-arounds we have in use. It would be nice to have the merging
> model in use, but it's not critical for us at the present.
>
> - Jesse
>
>
>
> On Mon, Apr 23, 2012 at 10:33 AM, Angela Spinazze <ats@atspin.com> wrote:
> > Hi all and thanks for the discussion about 2.4. I have cc'd the Talk
> list
> > so that everyone can see this discussion as it is precisely the type of
> > conversation that we want to have happen in our community space.
> >
> > What I take away from the discussion thread is that there is a tension
> > between taking on the merging work now versus shifting that work to a
> later
> > point in our development schedule so that the immediate needs of
> deployments
> > at UCB might instead be more efficiently addressed (based on current
> > resource constraints). The merging work has been talked about and pushed
> out
> > for over a year now so the landscape looks different from the deployer
> > perspective than it once did, yet the change to the model that the
> merging
> > work represents remains critical to the long-term viability of the
> software
> > for future deployers.
> >
> > A couple of questions that come to mind are:
> >
> > What do we lose by not doing the merging work now?
> > I think that the answer is (based on Patrick's comments below) that we
> lose
> > the ability to extend repeatable groups, and we lose support for
> > localization (neither of which we have now). Is that correct? Chris,
> you
> > pointed out that UCB has already addressed repeatable groups through a
> > work-around. Chris Pott has confirmed that he has had to do the same
> thing
> > at SMK and I suspect that MMI and Walker have had to as well (but I don't
> > know for certain- Jesse, Chris, Nate please chime in if you are so
> moved).
> > This means that while the merging work will make extending repeatable
> > groups easier, more efficient and effective in the future, current
> deployers
> > won't necessarily benefit from the change to the model right now. Same
> goes
> > for localization.
> >
> > The other work defined for 2.4 ensures that we gain support for preferred
> > and non-preferred terms and support for non-default vocabularies and is
> > defined as a low-risk, achievable set of deliverables for a 2-week
> > development sprint.
> >
> > What is prevented from being developed in releases 2.6, 2.7, and 2.8 if
> we
> > do not do the merging work now?
> > I'm not sure that I know what the answer is to this question. Would we,
> for
> > example, still be able to work on relationships between catalog records
> (AKA
> > multiple object handling) in 2.7 without the merging work? Is anything
> > related to media handling(2.6) or improving the upgrade process (2.6)
> > affected by not doing the merging work?
> >
> > If what we gain from pushing the merging work out to a later time in the
> > development schedule is more beneficial for all current deployers, then I
> > support the shift. It sounds like that is the case, but I need your help
> > Patrick with answers to the questions above, in order to fully understand
> > the ramifications.
> >
> > Angela
> >
> > On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
> >
> > Yes, this is all correct so far.
> >
> > We *could* take a messier approach with the PT/NPT changes. That would
> work,
> > but is a bit messier in a number of places. However, the messier
> approach is
> > well understood and low risk.
> >
> > We had been planning to take an approach that was cleaner, but required
> > support for merging.
> >
> > Yesterday, discussions about how we would actually implement the merging
> led
> > to some questions about how this would look in the database, and the
> > implications for report authoring. There was some surprise from our
> > deployers at how complex the so-called "merged" model would be in the DB.
> >
> > For SMK, it will mean that any report authoring will be more complex if
> they
> > are injecting language all over.
> >
> > Richard and I are feeling ambivalent about this. As software engineers,
> we
> > feel that the functionality should be there to support merging, and that
> it
> > would make the code for services and app cleaner, and would present a
> nice
> > model to the UI. However, being practical, and considering the additional
> > pain that this will entail for reporting, I am having some doubts.
> Knowing
> > how long it is taking me to get code working in the app layer, I am
> > concerned about the risk.
> >
> > The details:
> >
> > Let's say we want to add language to some repeating group of fields
> (let's
> > call it FooInfo). This will entail setting up specific fields for
> language
> > in an extension schema (it actually has to be something like
> > smk_FooInfoList/smk_FooInfo/language), some associated config in the app
> > layer, and the obvious UI work to present it to the user.
> >
> > What happens on the back-end is that the new language fields end up in a
> > separate table (smk_FooInfo) that the app layer and services have to keep
> > aligned with the FooInfo part of the common schema. The UI and app layer
> > should be able to ensure that if there are three rows in FooInfo, there
> will
> > be three rows in smk_FooInfo. This is the code I would have to write, and
> > it entails changes to the core of the app layer, so is very high risk.
> >
> > Issues:
> >
> > If someone uses the REST APIs or import and does not get this right,
> > everything will get hosed (e.g., SMK's language markers will end up on
> the
> > wrong row for the notes they were supposed to be associated with. There
> is
> > no reasonable way for services to even detect when an import or REST call
> > has screwed this up.
> > When someone wants to generate a report that includes the language (or
> > selects for the language) for a note, the query will get a little funky,
> > requiring more careful joins and constraints in what is already a complex
> > data model.
> >
> > ChrisH was surprised that the fields would not be merged into the common
> > table, until we explained that it cannot without changing the common
> model
> > in the code and for all the users. Even if we split up the repository to
> be
> > one per tenant, our code (a lot of the the REST stuff) would get
> confused by
> > not having a consistent definition of the schema, and so we have to add a
> > new table for every injected field.
> >
> > One alternative is to defer or just not support merging. Instead, we
> would
> > either:
> >
> > Tell folks to live with the model they have now, in which they have to
> > redefine the whole repeating structure everywhere they want to add a
> field.
> > Ugly, but well understood. The main objection to this after the amount of
> > typing, was that upgrades could be problematic. If we can stabilize the
> > schema and stop changing them so much (which we really need to do
> anyway),
> > this should not be so much of an issue.
> > OR
> > Be more liberal about adding new fields where folks need them, even if we
> > are not supporting them in the UI right away. E.g., adding Language to a
> > bunch of groups would cause us little pain, and then SMK could contribute
> > templates that use the language fields.
> >
> > Neither of these are great, and I do not mean to be glib in suggesting
> them.
> > Am trying to be practical and realistic about our current resource
> > situation.
> >
> > HTH - Patrick
> >
> >
> > ________________________________
> > From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
> > Sent: Friday, April 20, 2012 9:01 AM
> > To: Carly Bogen
> > Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM;
> Patrick
> > Schmitz
> > Subject: Re: Concerns about 2.4
> >
> > Thanks, Carly. Yes, you're right. Patrick has been calling this work
> > "merging" because the app layer needs to merge the extension fields into
> the
> > core group somehow. This would be needed for PT/NPT if the common set of
> > term fields were all in one shared schema. However, Patrick has
> > acknowledged that the "common" fields could be in their own
> > authority-specific schema. We just have extra work to try to make sure
> > those fields don't diverge in the future. I might not be translating
> this
> > correctly :-)
> >
> > Chris
> >
> > On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
> >
> > Hi Chris,
> >
> > Based on the design and scope discussions we had this week, preferred and
> > non-preferred terms and support for non-default vocabularies are the main
> > drivers for 2.4. I'm not sure where the concern about extending core
> field
> > groups came from, since it was not something we discussed. Is this
> related
> > to the merging issue? If so, it was my understanding that fixing the
> > merging issue was key to making PT and NPT and non-default vocabularies
> > work.
> >
> > Patrick, maybe you can weigh in on this and help me understand.
> >
> > Thanks,
> > Carly
> >
> > On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman <
> chris.hoffman@berkeley.edu>
> > wrote:
> >>
> >> Hi all,
> >>
> >> I'm really, really happy to see how much work the team has done to make
> >> 2.4 more realistic. However at UCB we still have some concerns that I
> want
> >> to bring up with you and other deployers.
> >>
> >> In talking with Patrick and Richard yesterday, I'm concerned that the
> work
> >> to enable deployers to extend core field groups is going to swamp other
> >> areas of work that for us are absolutely critical. PAHMA needs to
> launch
> >> with version 2.4, or they will have to extend TMS licensing (probably
> >> costing tens of thousands of dollars) and answer to campus auditors.
> >>
> >> We also want to add fields to existing groups, but in the meantime, we
> >> have created our own versions of those groups as needed. The work
> Patrick
> >> described to me is hard, risky, and has other implications for data
> loading
> >> and reporting (reporting will be very hard and the approach might have
> to
> >> change completely to point iReport at the App layer instead of at
> Postgres
> >> tables). Patrick is the only one who can do this work now, and he is
> >> concerned about being on the critical path for something like this. He
> is
> >> off for 3 weeks starting in early June.
> >>
> >> For PAHMA to launch with 2.4, they need the preferred/non-preferred term
> >> stuff, the support for multiple vocabularies within a single authority,
> and
> >> a fix to the dollar-sign import bug. There is also some significant
> >> customization work here that will need help from Patrick's team (e.g.,
> >> supporting range search on object numbers, "show me objects with numbers
> >> 91-202 through 91-292").
> >>
> >> I suspect field group extension is critical for SMK in particular.
> >> However, we really are on the edge of a major situation here.
> >>
> >> Thanks,
> >> Chris
> >
> >
> >
> >
> >
> > _______________________________________________
> > Talk mailing list
> > Talk@lists.collectionspace.org
> >
> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org
> >
>
--
__________________________________
Carl Goodman
Executive Director
MUSEUM OF THE MOVING IMAGE
36-01 35 Avenue, Astoria, NY 11106
movingimage.us 718 777 6800
PS
Patrick Schmitz
Tue, Apr 24, 2012 4:47 PM
I am not sure that there will be much in the way of an impact, other than
the documentation and support challenges inherent in the issues already
identified. There are counter-balancing costs, but I think the decision
calculus is not much different from the one we are facing for the
development tasks.
If we do not support merging:
The clients of the hoster may complain about the extra work to customize in
the cases when they want to add an element to a repeating group. If they are
a full service hoster, making these changes for a fee, then they have to do
extra work to define the extension.
Clients (and.or a full service hoster) will have a simpler
import/report/batch/export authoring experience.
If we do support merging:
The clients of the hoster may complain about the extra work and complexity
to author reports, batch processes, and exports, in the cases when they have
added an element to a repeating group. If they are a full service hoster,
making these changes for a fee, then they have to do extra work to support
the extension in reporting, batch, etc.
Clients using the REST API's will have to be cautioned to take care when
creating or updating any item that has a merged field, so as not to
destabilize the repository. The REST API clients will also have to figure
out how to merge the fields themselves, or they will have to use the
application-layer APIs to fetch data (as json, not XML) from the repo.
HTH - Patrick
From: Carl Goodman [mailto:cgoodman@movingimage.us]
Sent: Tuesday, April 24, 2012 7:03 AM
To: Jesse Martinez
Cc: Angela Spinazze; Patrick Schmitz; Chris Hoffman; Carly Bogen; Heather
Hart; CollectionSpace Talk List; David A. Greenbaum
Subject: Re: [Talk] Concerns about 2.4
Please answer this for me: what are the implications on an org/co that is
HOSTING the software for others? And will those others be adversely
impacted?
Thanks,
Carl
On Mon, Apr 23, 2012 at 6:23 PM, Jesse Martinez jmartinez@movingimage.us
wrote:
Here at MMI, we are also using a work-around for extending repeatable
groups in the schema. I'm fine with continuing the current work-around
model and having a known yet stable db structure even if the process
is "messy"; we only have a few instances of this work-around in use so
it's manageable from our perspective. (For other institutions this may
not be a luxury.) Like Patrick mentioned, the biggest concerns I have
with the current model are upgrades and schema changes to these
repeatable fields, but this is a minimal problem due to the dearth of
work-arounds we have in use. It would be nice to have the merging
model in use, but it's not critical for us at the present.
On Mon, Apr 23, 2012 at 10:33 AM, Angela Spinazze ats@atspin.com wrote:
Hi all and thanks for the discussion about 2.4. I have cc'd the Talk list
so that everyone can see this discussion as it is precisely the type of
conversation that we want to have happen in our community space.
What I take away from the discussion thread is that there is a tension
between taking on the merging work now versus shifting that work to a
point in our development schedule so that the immediate needs of
at UCB might instead be more efficiently addressed (based on current
resource constraints). The merging work has been talked about and pushed
for over a year now so the landscape looks different from the deployer
perspective than it once did, yet the change to the model that the merging
work represents remains critical to the long-term viability of the
for future deployers.
A couple of questions that come to mind are:
What do we lose by not doing the merging work now?
I think that the answer is (based on Patrick's comments below) that we
the ability to extend repeatable groups, and we lose support for
localization (neither of which we have now). Is that correct? Chris, you
pointed out that UCB has already addressed repeatable groups through a
work-around. Chris Pott has confirmed that he has had to do the same
at SMK and I suspect that MMI and Walker have had to as well (but I don't
know for certain- Jesse, Chris, Nate please chime in if you are so moved).
This means that while the merging work will make extending repeatable
groups easier, more efficient and effective in the future, current
won't necessarily benefit from the change to the model right now. Same
for localization.
The other work defined for 2.4 ensures that we gain support for preferred
and non-preferred terms and support for non-default vocabularies and is
defined as a low-risk, achievable set of deliverables for a 2-week
development sprint.
What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we
do not do the merging work now?
I'm not sure that I know what the answer is to this question. Would we,
example, still be able to work on relationships between catalog records
multiple object handling) in 2.7 without the merging work? Is anything
related to media handling(2.6) or improving the upgrade process (2.6)
affected by not doing the merging work?
If what we gain from pushing the merging work out to a later time in the
development schedule is more beneficial for all current deployers, then I
support the shift. It sounds like that is the case, but I need your help
Patrick with answers to the questions above, in order to fully understand
the ramifications.
Angela
On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
Yes, this is all correct so far.
We could take a messier approach with the PT/NPT changes. That would
but is a bit messier in a number of places. However, the messier approach
well understood and low risk.
We had been planning to take an approach that was cleaner, but required
support for merging.
Yesterday, discussions about how we would actually implement the merging
to some questions about how this would look in the database, and the
implications for report authoring. There was some surprise from our
deployers at how complex the so-called "merged" model would be in the DB.
For SMK, it will mean that any report authoring will be more complex if
are injecting language all over.
Richard and I are feeling ambivalent about this. As software engineers, we
feel that the functionality should be there to support merging, and that
would make the code for services and app cleaner, and would present a nice
model to the UI. However, being practical, and considering the additional
pain that this will entail for reporting, I am having some doubts. Knowing
how long it is taking me to get code working in the app layer, I am
concerned about the risk.
The details:
Let's say we want to add language to some repeating group of fields (let's
call it FooInfo). This will entail setting up specific fields for language
in an extension schema (it actually has to be something like
smk_FooInfoList/smk_FooInfo/language), some associated config in the app
layer, and the obvious UI work to present it to the user.
What happens on the back-end is that the new language fields end up in a
separate table (smk_FooInfo) that the app layer and services have to keep
aligned with the FooInfo part of the common schema. The UI and app layer
should be able to ensure that if there are three rows in FooInfo, there
be three rows in smk_FooInfo. This is the code I would have to write, and
it entails changes to the core of the app layer, so is very high risk.
Issues:
If someone uses the REST APIs or import and does not get this right,
everything will get hosed (e.g., SMK's language markers will end up on the
wrong row for the notes they were supposed to be associated with. There is
no reasonable way for services to even detect when an import or REST call
has screwed this up.
When someone wants to generate a report that includes the language (or
selects for the language) for a note, the query will get a little funky,
requiring more careful joins and constraints in what is already a complex
data model.
ChrisH was surprised that the fields would not be merged into the common
table, until we explained that it cannot without changing the common model
in the code and for all the users. Even if we split up the repository to
one per tenant, our code (a lot of the the REST stuff) would get confused
not having a consistent definition of the schema, and so we have to add a
new table for every injected field.
One alternative is to defer or just not support merging. Instead, we would
either:
Tell folks to live with the model they have now, in which they have to
redefine the whole repeating structure everywhere they want to add a
Ugly, but well understood. The main objection to this after the amount of
typing, was that upgrades could be problematic. If we can stabilize the
schema and stop changing them so much (which we really need to do anyway),
this should not be so much of an issue.
OR
Be more liberal about adding new fields where folks need them, even if we
are not supporting them in the UI right away. E.g., adding Language to a
bunch of groups would cause us little pain, and then SMK could contribute
templates that use the language fields.
Neither of these are great, and I do not mean to be glib in suggesting
Am trying to be practical and realistic about our current resource
situation.
HTH - Patrick
From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
Sent: Friday, April 20, 2012 9:01 AM
To: Carly Bogen
Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM;
Schmitz
Subject: Re: Concerns about 2.4
Thanks, Carly. Yes, you're right. Patrick has been calling this work
"merging" because the app layer needs to merge the extension fields into
core group somehow. This would be needed for PT/NPT if the common set of
term fields were all in one shared schema. However, Patrick has
acknowledged that the "common" fields could be in their own
authority-specific schema. We just have extra work to try to make sure
those fields don't diverge in the future. I might not be translating this
correctly :-)
Chris
On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
Hi Chris,
Based on the design and scope discussions we had this week, preferred and
non-preferred terms and support for non-default vocabularies are the main
drivers for 2.4. I'm not sure where the concern about extending core
groups came from, since it was not something we discussed. Is this
to the merging issue? If so, it was my understanding that fixing the
merging issue was key to making PT and NPT and non-default vocabularies
work.
Patrick, maybe you can weigh in on this and help me understand.
Thanks,
Carly
On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman
Hi all,
I'm really, really happy to see how much work the team has done to make
2.4 more realistic. However at UCB we still have some concerns that I
to bring up with you and other deployers.
In talking with Patrick and Richard yesterday, I'm concerned that the
to enable deployers to extend core field groups is going to swamp other
areas of work that for us are absolutely critical. PAHMA needs to launch
with version 2.4, or they will have to extend TMS licensing (probably
costing tens of thousands of dollars) and answer to campus auditors.
We also want to add fields to existing groups, but in the meantime, we
have created our own versions of those groups as needed. The work
described to me is hard, risky, and has other implications for data
and reporting (reporting will be very hard and the approach might have to
change completely to point iReport at the App layer instead of at
tables). Patrick is the only one who can do this work now, and he is
concerned about being on the critical path for something like this. He
off for 3 weeks starting in early June.
For PAHMA to launch with 2.4, they need the preferred/non-preferred term
stuff, the support for multiple vocabularies within a single authority,
a fix to the dollar-sign import bug. There is also some significant
customization work here that will need help from Patrick's team (e.g.,
supporting range search on object numbers, "show me objects with numbers
91-202 through 91-292").
I suspect field group extension is critical for SMK in particular.
However, we really are on the edge of a major situation here.
Thanks,
Chris
--
Carl Goodman
Executive Director
MUSEUM OF THE MOVING IMAGE
36-01 35 Avenue, Astoria, NY 11106
movingimage.us 718 777 6800
I am not sure that there will be much in the way of an impact, other than
the documentation and support challenges inherent in the issues already
identified. There are counter-balancing costs, but I think the decision
calculus is not much different from the one we are facing for the
development tasks.
If we do not support merging:
The clients of the hoster may complain about the extra work to customize in
the cases when they want to add an element to a repeating group. If they are
a full service hoster, making these changes for a fee, then they have to do
extra work to define the extension.
Clients (and.or a full service hoster) will have a simpler
import/report/batch/export authoring experience.
If we do support merging:
The clients of the hoster may complain about the extra work and complexity
to author reports, batch processes, and exports, in the cases when they have
added an element to a repeating group. If they are a full service hoster,
making these changes for a fee, then they have to do extra work to support
the extension in reporting, batch, etc.
Clients using the REST API's will have to be cautioned to take care when
creating or updating any item that has a merged field, so as not to
destabilize the repository. The REST API clients will also have to figure
out how to merge the fields themselves, or they will have to use the
application-layer APIs to fetch data (as json, not XML) from the repo.
HTH - Patrick
_____
From: Carl Goodman [mailto:cgoodman@movingimage.us]
Sent: Tuesday, April 24, 2012 7:03 AM
To: Jesse Martinez
Cc: Angela Spinazze; Patrick Schmitz; Chris Hoffman; Carly Bogen; Heather
Hart; CollectionSpace Talk List; David A. Greenbaum
Subject: Re: [Talk] Concerns about 2.4
Please answer this for me: what are the implications on an org/co that is
HOSTING the software for others? And will those others be adversely
impacted?
Thanks,
Carl
On Mon, Apr 23, 2012 at 6:23 PM, Jesse Martinez <jmartinez@movingimage.us>
wrote:
Here at MMI, we are also using a work-around for extending repeatable
groups in the schema. I'm fine with continuing the current work-around
model and having a known yet stable db structure even if the process
is "messy"; we only have a few instances of this work-around in use so
it's manageable from our perspective. (For other institutions this may
not be a luxury.) Like Patrick mentioned, the biggest concerns I have
with the current model are upgrades and schema changes to these
repeatable fields, but this is a minimal problem due to the dearth of
work-arounds we have in use. It would be nice to have the merging
model in use, but it's not critical for us at the present.
- Jesse
On Mon, Apr 23, 2012 at 10:33 AM, Angela Spinazze <ats@atspin.com> wrote:
> Hi all and thanks for the discussion about 2.4. I have cc'd the Talk list
> so that everyone can see this discussion as it is precisely the type of
> conversation that we want to have happen in our community space.
>
> What I take away from the discussion thread is that there is a tension
> between taking on the merging work now versus shifting that work to a
later
> point in our development schedule so that the immediate needs of
deployments
> at UCB might instead be more efficiently addressed (based on current
> resource constraints). The merging work has been talked about and pushed
out
> for over a year now so the landscape looks different from the deployer
> perspective than it once did, yet the change to the model that the merging
> work represents remains critical to the long-term viability of the
software
> for future deployers.
>
> A couple of questions that come to mind are:
>
> What do we lose by not doing the merging work now?
> I think that the answer is (based on Patrick's comments below) that we
lose
> the ability to extend repeatable groups, and we lose support for
> localization (neither of which we have now). Is that correct? Chris, you
> pointed out that UCB has already addressed repeatable groups through a
> work-around. Chris Pott has confirmed that he has had to do the same
thing
> at SMK and I suspect that MMI and Walker have had to as well (but I don't
> know for certain- Jesse, Chris, Nate please chime in if you are so moved).
> This means that while the merging work will make extending repeatable
> groups easier, more efficient and effective in the future, current
deployers
> won't necessarily benefit from the change to the model right now. Same
goes
> for localization.
>
> The other work defined for 2.4 ensures that we gain support for preferred
> and non-preferred terms and support for non-default vocabularies and is
> defined as a low-risk, achievable set of deliverables for a 2-week
> development sprint.
>
> What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we
> do not do the merging work now?
> I'm not sure that I know what the answer is to this question. Would we,
for
> example, still be able to work on relationships between catalog records
(AKA
> multiple object handling) in 2.7 without the merging work? Is anything
> related to media handling(2.6) or improving the upgrade process (2.6)
> affected by not doing the merging work?
>
> If what we gain from pushing the merging work out to a later time in the
> development schedule is more beneficial for all current deployers, then I
> support the shift. It sounds like that is the case, but I need your help
> Patrick with answers to the questions above, in order to fully understand
> the ramifications.
>
> Angela
>
> On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
>
> Yes, this is all correct so far.
>
> We *could* take a messier approach with the PT/NPT changes. That would
work,
> but is a bit messier in a number of places. However, the messier approach
is
> well understood and low risk.
>
> We had been planning to take an approach that was cleaner, but required
> support for merging.
>
> Yesterday, discussions about how we would actually implement the merging
led
> to some questions about how this would look in the database, and the
> implications for report authoring. There was some surprise from our
> deployers at how complex the so-called "merged" model would be in the DB.
>
> For SMK, it will mean that any report authoring will be more complex if
they
> are injecting language all over.
>
> Richard and I are feeling ambivalent about this. As software engineers, we
> feel that the functionality should be there to support merging, and that
it
> would make the code for services and app cleaner, and would present a nice
> model to the UI. However, being practical, and considering the additional
> pain that this will entail for reporting, I am having some doubts. Knowing
> how long it is taking me to get code working in the app layer, I am
> concerned about the risk.
>
> The details:
>
> Let's say we want to add language to some repeating group of fields (let's
> call it FooInfo). This will entail setting up specific fields for language
> in an extension schema (it actually has to be something like
> smk_FooInfoList/smk_FooInfo/language), some associated config in the app
> layer, and the obvious UI work to present it to the user.
>
> What happens on the back-end is that the new language fields end up in a
> separate table (smk_FooInfo) that the app layer and services have to keep
> aligned with the FooInfo part of the common schema. The UI and app layer
> should be able to ensure that if there are three rows in FooInfo, there
will
> be three rows in smk_FooInfo. This is the code I would have to write, and
> it entails changes to the core of the app layer, so is very high risk.
>
> Issues:
>
> If someone uses the REST APIs or import and does not get this right,
> everything will get hosed (e.g., SMK's language markers will end up on the
> wrong row for the notes they were supposed to be associated with. There is
> no reasonable way for services to even detect when an import or REST call
> has screwed this up.
> When someone wants to generate a report that includes the language (or
> selects for the language) for a note, the query will get a little funky,
> requiring more careful joins and constraints in what is already a complex
> data model.
>
> ChrisH was surprised that the fields would not be merged into the common
> table, until we explained that it cannot without changing the common model
> in the code and for all the users. Even if we split up the repository to
be
> one per tenant, our code (a lot of the the REST stuff) would get confused
by
> not having a consistent definition of the schema, and so we have to add a
> new table for every injected field.
>
> One alternative is to defer or just not support merging. Instead, we would
> either:
>
> Tell folks to live with the model they have now, in which they have to
> redefine the whole repeating structure everywhere they want to add a
field.
> Ugly, but well understood. The main objection to this after the amount of
> typing, was that upgrades could be problematic. If we can stabilize the
> schema and stop changing them so much (which we really need to do anyway),
> this should not be so much of an issue.
> OR
> Be more liberal about adding new fields where folks need them, even if we
> are not supporting them in the UI right away. E.g., adding Language to a
> bunch of groups would cause us little pain, and then SMK could contribute
> templates that use the language fields.
>
> Neither of these are great, and I do not mean to be glib in suggesting
them.
> Am trying to be practical and realistic about our current resource
> situation.
>
> HTH - Patrick
>
>
> ________________________________
> From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
> Sent: Friday, April 20, 2012 9:01 AM
> To: Carly Bogen
> Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM;
Patrick
> Schmitz
> Subject: Re: Concerns about 2.4
>
> Thanks, Carly. Yes, you're right. Patrick has been calling this work
> "merging" because the app layer needs to merge the extension fields into
the
> core group somehow. This would be needed for PT/NPT if the common set of
> term fields were all in one shared schema. However, Patrick has
> acknowledged that the "common" fields could be in their own
> authority-specific schema. We just have extra work to try to make sure
> those fields don't diverge in the future. I might not be translating this
> correctly :-)
>
> Chris
>
> On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
>
> Hi Chris,
>
> Based on the design and scope discussions we had this week, preferred and
> non-preferred terms and support for non-default vocabularies are the main
> drivers for 2.4. I'm not sure where the concern about extending core
field
> groups came from, since it was not something we discussed. Is this
related
> to the merging issue? If so, it was my understanding that fixing the
> merging issue was key to making PT and NPT and non-default vocabularies
> work.
>
> Patrick, maybe you can weigh in on this and help me understand.
>
> Thanks,
> Carly
>
> On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman
<chris.hoffman@berkeley.edu>
> wrote:
>>
>> Hi all,
>>
>> I'm really, really happy to see how much work the team has done to make
>> 2.4 more realistic. However at UCB we still have some concerns that I
want
>> to bring up with you and other deployers.
>>
>> In talking with Patrick and Richard yesterday, I'm concerned that the
work
>> to enable deployers to extend core field groups is going to swamp other
>> areas of work that for us are absolutely critical. PAHMA needs to launch
>> with version 2.4, or they will have to extend TMS licensing (probably
>> costing tens of thousands of dollars) and answer to campus auditors.
>>
>> We also want to add fields to existing groups, but in the meantime, we
>> have created our own versions of those groups as needed. The work
Patrick
>> described to me is hard, risky, and has other implications for data
loading
>> and reporting (reporting will be very hard and the approach might have to
>> change completely to point iReport at the App layer instead of at
Postgres
>> tables). Patrick is the only one who can do this work now, and he is
>> concerned about being on the critical path for something like this. He
is
>> off for 3 weeks starting in early June.
>>
>> For PAHMA to launch with 2.4, they need the preferred/non-preferred term
>> stuff, the support for multiple vocabularies within a single authority,
and
>> a fix to the dollar-sign import bug. There is also some significant
>> customization work here that will need help from Patrick's team (e.g.,
>> supporting range search on object numbers, "show me objects with numbers
>> 91-202 through 91-292").
>>
>> I suspect field group extension is critical for SMK in particular.
>> However, we really are on the edge of a major situation here.
>>
>> Thanks,
>> Chris
>
>
>
>
>
> _______________________________________________
> Talk mailing list
> Talk@lists.collectionspace.org
>
http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace
.org
>
--
__________________________________
Carl Goodman
Executive Director
MUSEUM OF THE MOVING IMAGE
36-01 35 Avenue, Astoria, NY 11106
movingimage.us 718 777 6800
AS
Angela Spinazze
Tue, Apr 24, 2012 9:24 PM
I would say that at this point, given what we know and the implications of supporting and not supporting merging, that we could move this work to a point later in our schedule - meaning out of 2.4.
Patrick, re: your 2 additional points from yesterday:
Two additional points:
-
2.4 will take more than two weeks in any case.
I am a bit confused about why this is still the case. My recollection was that we added the extra week to account for the merging work. Now that we're removing it, you're suggesting that we still need 3 weeks to do the remaining work - is that correct?
-
We should seriously consider the suggestion to add some additional fields into our common schemas to mitigate the problems ChrisP et al. are facing. I am not saying we do this quickly without discussion, as any changes to the schemas need discussion. However, we can allow many changes to the schemas (like adding language markers, or additional notes fields) without requiring that the demo UI we ship expose all those fields. If this makes sense, it might be a useful compromise.
This does seem like a useful compromise, and I would suggest that we discuss this as part of the 2.8 timeframe. We will need to document the requirements in order to determine the scope of this work. Or, point me to something that already exists, if I'm just not remembering well enough where it is.
Angela
On Apr 24, 2012, at 11:47 AM, Patrick Schmitz wrote:
I am not sure that there will be much in the way of an impact, other than the documentation and support challenges inherent in the issues already identified. There are counter-balancing costs, but I think the decision calculus is not much different from the one we are facing for the development tasks.
If we do not support merging:
The clients of the hoster may complain about the extra work to customize in the cases when they want to add an element to a repeating group. If they are a full service hoster, making these changes for a fee, then they have to do extra work to define the extension.
Clients (and.or a full service hoster) will have a simpler import/report/batch/export authoring experience.
If we do support merging:
The clients of the hoster may complain about the extra work and complexity to author reports, batch processes, and exports, in the cases when they have added an element to a repeating group. If they are a full service hoster, making these changes for a fee, then they have to do extra work to support the extension in reporting, batch, etc.
Clients using the REST API's will have to be cautioned to take care when creating or updating any item that has a merged field, so as not to destabilize the repository. The REST API clients will also have to figure out how to merge the fields themselves, or they will have to use the application-layer APIs to fetch data (as json, not XML) from the repo.
HTH - Patrick
From: Carl Goodman [mailto:cgoodman@movingimage.us]
Sent: Tuesday, April 24, 2012 7:03 AM
To: Jesse Martinez
Cc: Angela Spinazze; Patrick Schmitz; Chris Hoffman; Carly Bogen; Heather Hart; CollectionSpace Talk List; David A. Greenbaum
Subject: Re: [Talk] Concerns about 2.4
Please answer this for me: what are the implications on an org/co that is HOSTING the software for others? And will those others be adversely impacted?
Thanks,
Carl
On Mon, Apr 23, 2012 at 6:23 PM, Jesse Martinez jmartinez@movingimage.us wrote:
Here at MMI, we are also using a work-around for extending repeatable
groups in the schema. I'm fine with continuing the current work-around
model and having a known yet stable db structure even if the process
is "messy"; we only have a few instances of this work-around in use so
it's manageable from our perspective. (For other institutions this may
not be a luxury.) Like Patrick mentioned, the biggest concerns I have
with the current model are upgrades and schema changes to these
repeatable fields, but this is a minimal problem due to the dearth of
work-arounds we have in use. It would be nice to have the merging
model in use, but it's not critical for us at the present.
On Mon, Apr 23, 2012 at 10:33 AM, Angela Spinazze ats@atspin.com wrote:
Hi all and thanks for the discussion about 2.4. I have cc'd the Talk list
so that everyone can see this discussion as it is precisely the type of
conversation that we want to have happen in our community space.
What I take away from the discussion thread is that there is a tension
between taking on the merging work now versus shifting that work to a later
point in our development schedule so that the immediate needs of deployments
at UCB might instead be more efficiently addressed (based on current
resource constraints). The merging work has been talked about and pushed out
for over a year now so the landscape looks different from the deployer
perspective than it once did, yet the change to the model that the merging
work represents remains critical to the long-term viability of the software
for future deployers.
A couple of questions that come to mind are:
What do we lose by not doing the merging work now?
I think that the answer is (based on Patrick's comments below) that we lose
the ability to extend repeatable groups, and we lose support for
localization (neither of which we have now). Is that correct? Chris, you
pointed out that UCB has already addressed repeatable groups through a
work-around. Chris Pott has confirmed that he has had to do the same thing
at SMK and I suspect that MMI and Walker have had to as well (but I don't
know for certain- Jesse, Chris, Nate please chime in if you are so moved).
This means that while the merging work will make extending repeatable
groups easier, more efficient and effective in the future, current deployers
won't necessarily benefit from the change to the model right now. Same goes
for localization.
The other work defined for 2.4 ensures that we gain support for preferred
and non-preferred terms and support for non-default vocabularies and is
defined as a low-risk, achievable set of deliverables for a 2-week
development sprint.
What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we
do not do the merging work now?
I'm not sure that I know what the answer is to this question. Would we, for
example, still be able to work on relationships between catalog records (AKA
multiple object handling) in 2.7 without the merging work? Is anything
related to media handling(2.6) or improving the upgrade process (2.6)
affected by not doing the merging work?
If what we gain from pushing the merging work out to a later time in the
development schedule is more beneficial for all current deployers, then I
support the shift. It sounds like that is the case, but I need your help
Patrick with answers to the questions above, in order to fully understand
the ramifications.
Angela
On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
Yes, this is all correct so far.
We could take a messier approach with the PT/NPT changes. That would work,
but is a bit messier in a number of places. However, the messier approach is
well understood and low risk.
We had been planning to take an approach that was cleaner, but required
support for merging.
Yesterday, discussions about how we would actually implement the merging led
to some questions about how this would look in the database, and the
implications for report authoring. There was some surprise from our
deployers at how complex the so-called "merged" model would be in the DB.
For SMK, it will mean that any report authoring will be more complex if they
are injecting language all over.
Richard and I are feeling ambivalent about this. As software engineers, we
feel that the functionality should be there to support merging, and that it
would make the code for services and app cleaner, and would present a nice
model to the UI. However, being practical, and considering the additional
pain that this will entail for reporting, I am having some doubts. Knowing
how long it is taking me to get code working in the app layer, I am
concerned about the risk.
The details:
Let's say we want to add language to some repeating group of fields (let's
call it FooInfo). This will entail setting up specific fields for language
in an extension schema (it actually has to be something like
smk_FooInfoList/smk_FooInfo/language), some associated config in the app
layer, and the obvious UI work to present it to the user.
What happens on the back-end is that the new language fields end up in a
separate table (smk_FooInfo) that the app layer and services have to keep
aligned with the FooInfo part of the common schema. The UI and app layer
should be able to ensure that if there are three rows in FooInfo, there will
be three rows in smk_FooInfo. This is the code I would have to write, and
it entails changes to the core of the app layer, so is very high risk.
Issues:
If someone uses the REST APIs or import and does not get this right,
everything will get hosed (e.g., SMK's language markers will end up on the
wrong row for the notes they were supposed to be associated with. There is
no reasonable way for services to even detect when an import or REST call
has screwed this up.
When someone wants to generate a report that includes the language (or
selects for the language) for a note, the query will get a little funky,
requiring more careful joins and constraints in what is already a complex
data model.
ChrisH was surprised that the fields would not be merged into the common
table, until we explained that it cannot without changing the common model
in the code and for all the users. Even if we split up the repository to be
one per tenant, our code (a lot of the the REST stuff) would get confused by
not having a consistent definition of the schema, and so we have to add a
new table for every injected field.
One alternative is to defer or just not support merging. Instead, we would
either:
Tell folks to live with the model they have now, in which they have to
redefine the whole repeating structure everywhere they want to add a field.
Ugly, but well understood. The main objection to this after the amount of
typing, was that upgrades could be problematic. If we can stabilize the
schema and stop changing them so much (which we really need to do anyway),
this should not be so much of an issue.
OR
Be more liberal about adding new fields where folks need them, even if we
are not supporting them in the UI right away. E.g., adding Language to a
bunch of groups would cause us little pain, and then SMK could contribute
templates that use the language fields.
Neither of these are great, and I do not mean to be glib in suggesting them.
Am trying to be practical and realistic about our current resource
situation.
HTH - Patrick
From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
Sent: Friday, April 20, 2012 9:01 AM
To: Carly Bogen
Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM; Patrick
Schmitz
Subject: Re: Concerns about 2.4
Thanks, Carly. Yes, you're right. Patrick has been calling this work
"merging" because the app layer needs to merge the extension fields into the
core group somehow. This would be needed for PT/NPT if the common set of
term fields were all in one shared schema. However, Patrick has
acknowledged that the "common" fields could be in their own
authority-specific schema. We just have extra work to try to make sure
those fields don't diverge in the future. I might not be translating this
correctly :-)
Chris
On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
Hi Chris,
Based on the design and scope discussions we had this week, preferred and
non-preferred terms and support for non-default vocabularies are the main
drivers for 2.4. I'm not sure where the concern about extending core field
groups came from, since it was not something we discussed. Is this related
to the merging issue? If so, it was my understanding that fixing the
merging issue was key to making PT and NPT and non-default vocabularies
work.
Patrick, maybe you can weigh in on this and help me understand.
Thanks,
Carly
On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman chris.hoffman@berkeley.edu
wrote:
Hi all,
I'm really, really happy to see how much work the team has done to make
2.4 more realistic. However at UCB we still have some concerns that I want
to bring up with you and other deployers.
In talking with Patrick and Richard yesterday, I'm concerned that the work
to enable deployers to extend core field groups is going to swamp other
areas of work that for us are absolutely critical. PAHMA needs to launch
with version 2.4, or they will have to extend TMS licensing (probably
costing tens of thousands of dollars) and answer to campus auditors.
We also want to add fields to existing groups, but in the meantime, we
have created our own versions of those groups as needed. The work Patrick
described to me is hard, risky, and has other implications for data loading
and reporting (reporting will be very hard and the approach might have to
change completely to point iReport at the App layer instead of at Postgres
tables). Patrick is the only one who can do this work now, and he is
concerned about being on the critical path for something like this. He is
off for 3 weeks starting in early June.
For PAHMA to launch with 2.4, they need the preferred/non-preferred term
stuff, the support for multiple vocabularies within a single authority, and
a fix to the dollar-sign import bug. There is also some significant
customization work here that will need help from Patrick's team (e.g.,
supporting range search on object numbers, "show me objects with numbers
91-202 through 91-292").
I suspect field group extension is critical for SMK in particular.
However, we really are on the edge of a major situation here.
Thanks,
Chris
--
Carl Goodman
Executive Director
MUSEUM OF THE MOVING IMAGE
36-01 35 Avenue, Astoria, NY 11106
movingimage.us 718 777 6800
I would say that at this point, given what we know and the implications of supporting and not supporting merging, that we could move this work to a point later in our schedule - meaning out of 2.4.
Patrick, re: your 2 additional points from yesterday:
Two additional points:
1. 2.4 will take more than two weeks in any case.
I am a bit confused about why this is still the case. My recollection was that we added the extra week to account for the merging work. Now that we're removing it, you're suggesting that we still need 3 weeks to do the remaining work - is that correct?
2. We should seriously consider the suggestion to add some additional fields into our common schemas to mitigate the problems ChrisP et al. are facing. I am not saying we do this quickly without discussion, as any changes to the schemas need discussion. However, we can allow many changes to the schemas (like adding language markers, or additional notes fields) without requiring that the demo UI we ship expose all those fields. If this makes sense, it might be a useful compromise.
This does seem like a useful compromise, and I would suggest that we discuss this as part of the 2.8 timeframe. We will need to document the requirements in order to determine the scope of this work. Or, point me to something that already exists, if I'm just not remembering well enough where it is.
Angela
On Apr 24, 2012, at 11:47 AM, Patrick Schmitz wrote:
> I am not sure that there will be much in the way of an impact, other than the documentation and support challenges inherent in the issues already identified. There are counter-balancing costs, but I think the decision calculus is not much different from the one we are facing for the development tasks.
>
> If we do not support merging:
>
> The clients of the hoster may complain about the extra work to customize in the cases when they want to add an element to a repeating group. If they are a full service hoster, making these changes for a fee, then they have to do extra work to define the extension.
>
> Clients (and.or a full service hoster) will have a simpler import/report/batch/export authoring experience.
>
> If we do support merging:
>
> The clients of the hoster may complain about the extra work and complexity to author reports, batch processes, and exports, in the cases when they have added an element to a repeating group. If they are a full service hoster, making these changes for a fee, then they have to do extra work to support the extension in reporting, batch, etc.
>
> Clients using the REST API's will have to be cautioned to take care when creating or updating any item that has a merged field, so as not to destabilize the repository. The REST API clients will also have to figure out how to merge the fields themselves, or they will have to use the application-layer APIs to fetch data (as json, not XML) from the repo.
>
>
> HTH - Patrick
>
> From: Carl Goodman [mailto:cgoodman@movingimage.us]
> Sent: Tuesday, April 24, 2012 7:03 AM
> To: Jesse Martinez
> Cc: Angela Spinazze; Patrick Schmitz; Chris Hoffman; Carly Bogen; Heather Hart; CollectionSpace Talk List; David A. Greenbaum
> Subject: Re: [Talk] Concerns about 2.4
>
> Please answer this for me: what are the implications on an org/co that is HOSTING the software for others? And will those others be adversely impacted?
>
> Thanks,
> Carl
>
> On Mon, Apr 23, 2012 at 6:23 PM, Jesse Martinez <jmartinez@movingimage.us> wrote:
> Here at MMI, we are also using a work-around for extending repeatable
> groups in the schema. I'm fine with continuing the current work-around
> model and having a known yet stable db structure even if the process
> is "messy"; we only have a few instances of this work-around in use so
> it's manageable from our perspective. (For other institutions this may
> not be a luxury.) Like Patrick mentioned, the biggest concerns I have
> with the current model are upgrades and schema changes to these
> repeatable fields, but this is a minimal problem due to the dearth of
> work-arounds we have in use. It would be nice to have the merging
> model in use, but it's not critical for us at the present.
>
> - Jesse
>
>
>
> On Mon, Apr 23, 2012 at 10:33 AM, Angela Spinazze <ats@atspin.com> wrote:
> > Hi all and thanks for the discussion about 2.4. I have cc'd the Talk list
> > so that everyone can see this discussion as it is precisely the type of
> > conversation that we want to have happen in our community space.
> >
> > What I take away from the discussion thread is that there is a tension
> > between taking on the merging work now versus shifting that work to a later
> > point in our development schedule so that the immediate needs of deployments
> > at UCB might instead be more efficiently addressed (based on current
> > resource constraints). The merging work has been talked about and pushed out
> > for over a year now so the landscape looks different from the deployer
> > perspective than it once did, yet the change to the model that the merging
> > work represents remains critical to the long-term viability of the software
> > for future deployers.
> >
> > A couple of questions that come to mind are:
> >
> > What do we lose by not doing the merging work now?
> > I think that the answer is (based on Patrick's comments below) that we lose
> > the ability to extend repeatable groups, and we lose support for
> > localization (neither of which we have now). Is that correct? Chris, you
> > pointed out that UCB has already addressed repeatable groups through a
> > work-around. Chris Pott has confirmed that he has had to do the same thing
> > at SMK and I suspect that MMI and Walker have had to as well (but I don't
> > know for certain- Jesse, Chris, Nate please chime in if you are so moved).
> > This means that while the merging work will make extending repeatable
> > groups easier, more efficient and effective in the future, current deployers
> > won't necessarily benefit from the change to the model right now. Same goes
> > for localization.
> >
> > The other work defined for 2.4 ensures that we gain support for preferred
> > and non-preferred terms and support for non-default vocabularies and is
> > defined as a low-risk, achievable set of deliverables for a 2-week
> > development sprint.
> >
> > What is prevented from being developed in releases 2.6, 2.7, and 2.8 if we
> > do not do the merging work now?
> > I'm not sure that I know what the answer is to this question. Would we, for
> > example, still be able to work on relationships between catalog records (AKA
> > multiple object handling) in 2.7 without the merging work? Is anything
> > related to media handling(2.6) or improving the upgrade process (2.6)
> > affected by not doing the merging work?
> >
> > If what we gain from pushing the merging work out to a later time in the
> > development schedule is more beneficial for all current deployers, then I
> > support the shift. It sounds like that is the case, but I need your help
> > Patrick with answers to the questions above, in order to fully understand
> > the ramifications.
> >
> > Angela
> >
> > On Apr 20, 2012, at 12:03 PM, Patrick Schmitz wrote:
> >
> > Yes, this is all correct so far.
> >
> > We *could* take a messier approach with the PT/NPT changes. That would work,
> > but is a bit messier in a number of places. However, the messier approach is
> > well understood and low risk.
> >
> > We had been planning to take an approach that was cleaner, but required
> > support for merging.
> >
> > Yesterday, discussions about how we would actually implement the merging led
> > to some questions about how this would look in the database, and the
> > implications for report authoring. There was some surprise from our
> > deployers at how complex the so-called "merged" model would be in the DB.
> >
> > For SMK, it will mean that any report authoring will be more complex if they
> > are injecting language all over.
> >
> > Richard and I are feeling ambivalent about this. As software engineers, we
> > feel that the functionality should be there to support merging, and that it
> > would make the code for services and app cleaner, and would present a nice
> > model to the UI. However, being practical, and considering the additional
> > pain that this will entail for reporting, I am having some doubts. Knowing
> > how long it is taking me to get code working in the app layer, I am
> > concerned about the risk.
> >
> > The details:
> >
> > Let's say we want to add language to some repeating group of fields (let's
> > call it FooInfo). This will entail setting up specific fields for language
> > in an extension schema (it actually has to be something like
> > smk_FooInfoList/smk_FooInfo/language), some associated config in the app
> > layer, and the obvious UI work to present it to the user.
> >
> > What happens on the back-end is that the new language fields end up in a
> > separate table (smk_FooInfo) that the app layer and services have to keep
> > aligned with the FooInfo part of the common schema. The UI and app layer
> > should be able to ensure that if there are three rows in FooInfo, there will
> > be three rows in smk_FooInfo. This is the code I would have to write, and
> > it entails changes to the core of the app layer, so is very high risk.
> >
> > Issues:
> >
> > If someone uses the REST APIs or import and does not get this right,
> > everything will get hosed (e.g., SMK's language markers will end up on the
> > wrong row for the notes they were supposed to be associated with. There is
> > no reasonable way for services to even detect when an import or REST call
> > has screwed this up.
> > When someone wants to generate a report that includes the language (or
> > selects for the language) for a note, the query will get a little funky,
> > requiring more careful joins and constraints in what is already a complex
> > data model.
> >
> > ChrisH was surprised that the fields would not be merged into the common
> > table, until we explained that it cannot without changing the common model
> > in the code and for all the users. Even if we split up the repository to be
> > one per tenant, our code (a lot of the the REST stuff) would get confused by
> > not having a consistent definition of the schema, and so we have to add a
> > new table for every injected field.
> >
> > One alternative is to defer or just not support merging. Instead, we would
> > either:
> >
> > Tell folks to live with the model they have now, in which they have to
> > redefine the whole repeating structure everywhere they want to add a field.
> > Ugly, but well understood. The main objection to this after the amount of
> > typing, was that upgrades could be problematic. If we can stabilize the
> > schema and stop changing them so much (which we really need to do anyway),
> > this should not be so much of an issue.
> > OR
> > Be more liberal about adding new fields where folks need them, even if we
> > are not supporting them in the UI right away. E.g., adding Language to a
> > bunch of groups would cause us little pain, and then SMK could contribute
> > templates that use the language fields.
> >
> > Neither of these are great, and I do not mean to be glib in suggesting them.
> > Am trying to be practical and realistic about our current resource
> > situation.
> >
> > HTH - Patrick
> >
> >
> > ________________________________
> > From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
> > Sent: Friday, April 20, 2012 9:01 AM
> > To: Carly Bogen
> > Cc: Angela Spinazze; Heather Hart; Carl Goodman; David A. GREENBAUM; Patrick
> > Schmitz
> > Subject: Re: Concerns about 2.4
> >
> > Thanks, Carly. Yes, you're right. Patrick has been calling this work
> > "merging" because the app layer needs to merge the extension fields into the
> > core group somehow. This would be needed for PT/NPT if the common set of
> > term fields were all in one shared schema. However, Patrick has
> > acknowledged that the "common" fields could be in their own
> > authority-specific schema. We just have extra work to try to make sure
> > those fields don't diverge in the future. I might not be translating this
> > correctly :-)
> >
> > Chris
> >
> > On Apr 20, 2012, at 8:54 AM, Carly Bogen wrote:
> >
> > Hi Chris,
> >
> > Based on the design and scope discussions we had this week, preferred and
> > non-preferred terms and support for non-default vocabularies are the main
> > drivers for 2.4. I'm not sure where the concern about extending core field
> > groups came from, since it was not something we discussed. Is this related
> > to the merging issue? If so, it was my understanding that fixing the
> > merging issue was key to making PT and NPT and non-default vocabularies
> > work.
> >
> > Patrick, maybe you can weigh in on this and help me understand.
> >
> > Thanks,
> > Carly
> >
> > On Fri, Apr 20, 2012 at 11:50 AM, Chris Hoffman <chris.hoffman@berkeley.edu>
> > wrote:
> >>
> >> Hi all,
> >>
> >> I'm really, really happy to see how much work the team has done to make
> >> 2.4 more realistic. However at UCB we still have some concerns that I want
> >> to bring up with you and other deployers.
> >>
> >> In talking with Patrick and Richard yesterday, I'm concerned that the work
> >> to enable deployers to extend core field groups is going to swamp other
> >> areas of work that for us are absolutely critical. PAHMA needs to launch
> >> with version 2.4, or they will have to extend TMS licensing (probably
> >> costing tens of thousands of dollars) and answer to campus auditors.
> >>
> >> We also want to add fields to existing groups, but in the meantime, we
> >> have created our own versions of those groups as needed. The work Patrick
> >> described to me is hard, risky, and has other implications for data loading
> >> and reporting (reporting will be very hard and the approach might have to
> >> change completely to point iReport at the App layer instead of at Postgres
> >> tables). Patrick is the only one who can do this work now, and he is
> >> concerned about being on the critical path for something like this. He is
> >> off for 3 weeks starting in early June.
> >>
> >> For PAHMA to launch with 2.4, they need the preferred/non-preferred term
> >> stuff, the support for multiple vocabularies within a single authority, and
> >> a fix to the dollar-sign import bug. There is also some significant
> >> customization work here that will need help from Patrick's team (e.g.,
> >> supporting range search on object numbers, "show me objects with numbers
> >> 91-202 through 91-292").
> >>
> >> I suspect field group extension is critical for SMK in particular.
> >> However, we really are on the edge of a major situation here.
> >>
> >> Thanks,
> >> Chris
> >
> >
> >
> >
> >
> > _______________________________________________
> > Talk mailing list
> > Talk@lists.collectionspace.org
> > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org
> >
>
>
>
> --
>
> __________________________________
> Carl Goodman
> Executive Director
>
> MUSEUM OF THE MOVING IMAGE
> 36-01 35 Avenue, Astoria, NY 11106
> movingimage.us 718 777 6800
>
