talk@lists.collectionspace.org

WE HAVE SUNSET THIS LISTSERV - Join us at collectionspace@lyrasislists.org

View all threads

what's the best way to represent structured dates?

PS
Patrick Schmitz
Wed, Aug 24, 2011 5:23 PM

Thanks, Susan.

PostgreSQL date supports the date range 4713 BC to 5874897 AD. Personally, I
wish they had balanced past dates better, but such is life.

MySQL is more restrictive, and only supports 1000 AD to 9999 AD. Sigh.

It appears that we will have to implement a custom data type for our dates.
A Java Gregorian calendar  (which can be converted to a signed long, and
stored in the DB as such) can represent dates back to almost 300 Million
years BCE (earliest year is 292269055 BCE). Would this be enough range? If
not, we'd have to do something ever funkier.

Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf Of
Susan Stone
Sent: Wednesday, August 24, 2011 9:34 AM
To: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent structured dates?

Patrick,

There was quite a bit of discussion of this in our Wednesday
afternoon meetings at Berkeley that I'm sure Aron will
recall. I don't think date fields go back far enough in time,
so I'd recommend using integers or floating point numbers and
providing the math in search. It is also important to make
sure that false precision isn't introduced. If something is
dated 1900, the system shouldn't enter that as January 1,
1900 unless the uncertainty values can be used to
differentiate and the display date can be simply 1900.

Susan

On 08/24/2011 09:16 AM, Patrick Schmitz wrote:

Really? I am a little surprised that in all the design

discussions for

structured dates this did not come up.
Anyway, I will say that we do not have the time to build

some system

that allows for a lot of custom behavior, or elaborate

heuristics here

(i.e., trying to divine what is intended from a set of text

fields). I

would strongly suggest that we either

  1. require entry of some actual date fields upon which the

application

 can reason,
 or
  1. we define a very simple set of rules for building such

date values

 from some of the existing text fields. By very simple, 

I mean that

 we only consider fields like year, month, day, and that 

the rules

 produce two scalar dates (early and late), which either define a
 range, or define a point (if they are equal). AIUI, a structured
 date /cannot/ model the idea of "some time/any time after (or
 before) a date".

One further question is the semantic of which date (if any) will be
used for ordering.
If and when advanced search supports date ranges (which

also implies

support for /before/ and /after/ logic in search), we can

handle the

interval logic associated with the structured dates, but only if we
have a pair of scalar date values.
Down the road, I do think it would be interesting to implement some
fuzzier logic on date search, to better support "circa"

logic and the

uncertainty values. However, I suspect that it will take longer to
arrive at consensus on these semantics, and would defer that
discussion for now.
Thanks - Patrick



 *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
 *Sent:* Wednesday, August 24, 2011 9:01 AM
 *To:* Patrick Schmitz
 *Cc:* 'Christopher Pott'; 'Aron Roberts'; 
 *Subject:* Re: [Talk] what's the best way to represent 

structured dates?

 Hi Patrick, I think it's fair to say there's no 

consensus because

 there's been no discussion :-) It's possible that each 

customer will

 want to use structured dates differently, which will 

make it hard to

 add any intelligence or even predictability to things 

like search.

 However, I also think that if a significant number of deploying
 institutions can come to some consensus on best ways to use and
 search structured dates, then many institutions will 

choose to adopt

 the same best practices.

 Thanks,
 Chris

 On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote:
 Aron was out yesterday, and Rick is out for a few 

days. They have

 done a lot of the structured dates work to date.
 The last I heard, Aron was hoping to get consensus 

from the users

 about how we map all the text fields to actual date 

fields that we

 can reason on (for search, ordering, etc). To date, 

there seems to

 be no such consensus, which will preclude search, 

ordering, etc.

 on structured dates. I would think that this would 

make structured

 dates quite a bit less useful...
 Perhaps (I hope) my understanding is out of date, and 

we just need

 to implement the logic. Perhaps Aron can jump in this 

AM, when he

 gets in.
 Patrick

     


     *From:* talk-bounces@lists.collectionspace.org
     <mailto:talk-bounces@lists.collectionspace.org>
     [mailto:talk-bounces@lists.collectionspace.org] 

*On Behalf Of

     *Chris Hoffman
     *Sent:* Tuesday, August 23, 2011 11:05 AM
     *To:* Christopher Pott
     *Cc:* talk@lists.collectionspace.org
     <mailto:talk@lists.collectionspace.org>
     *Subject:* Re: [Talk] what's the best way to represent
     structured dates?

     Hi Chris,

     Thanks for asking these questions. We've been wondering the
     same things. Have you changed any existing calendar date
     fields to structured dates or added new structured date
     fields? I think this is all quite new, and 

together we might

     be able to come up with some best practices 

(related to data

     migration and data entry).

     I suspect that search and advanced search of 

structured date

     fields is going to be an issue for those of us deploying
     CollectionSpace. Our users will want CSpace to be 

smart about

     structured dates, interpreting qualifiers, and so on, but I
     bet that we won't see much as much as we would like in the
     first version of advanced search. I'm always happy to be
     proved wrong! :-)

     Chris

     On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote:
     Hi,

     I'm looking at the structured date schema and the 

best way to

     represent dates migrated from our old system. Any 

information

     about the following would be much appreciated..

     1. I see some different/overlapping possibilities for
     recording an object creation date. For example, 

to represent

     "1718" (display date) one could either store:

     a) a single date, providing year only (1718)

     b) an earliest date (01.01.1718) with a latest date 

(31.12.1718)

     c) a single date (01.01.1718) with a qualifier of 
  • 364 days
     Is there any difference in the meaning of these three (and
     would they all be treated equally by services such as
     advanced search)? Our migration data is currently 

in format

     'b' but the other formats are sometimes 

'friendlier' for new

     input.

     2. How are date "certainties" and "qualifiers" used by
     advanced search?

     I've specified "1718", as a creation date with a 

certainty of

     "circa". Will a search for objects created during 

1717 result

     in a hit?

     Or is it necessary to use qualifiers (eg. 1718 

+/- 2 years)

     to define exact limits for this type of searching?

     3. Is the Date schema locally extendable? (We 

need a field to

     hold an English translation of the display text)

     4. Will Advanced search by creation date also search the
     display date or will it just use the single/earliest and
     latest date groups?

     Thanks,

     Chris

     _______________________________________________
     Talk mailing list
     Talk@lists.collectionspace.org
     <mailto:Talk@lists.collectionspace.org>
     

ionspace.org


Talk mailing list
Talk@lists.collectionspace.org

nspace.org

Thanks, Susan. PostgreSQL date supports the date range 4713 BC to 5874897 AD. Personally, I wish they had balanced past dates better, but such is life. MySQL is more restrictive, and only supports 1000 AD to 9999 AD. Sigh. It appears that we will have to implement a custom data type for our dates. A Java Gregorian calendar (which can be converted to a signed long, and stored in the DB as such) can represent dates back to almost 300 Million years BCE (earliest year is 292269055 BCE). Would this be enough range? If not, we'd have to do something ever funkier. Patrick > -----Original Message----- > From: talk-bounces@lists.collectionspace.org > [mailto:talk-bounces@lists.collectionspace.org] On Behalf Of > Susan Stone > Sent: Wednesday, August 24, 2011 9:34 AM > To: talk@lists.collectionspace.org > Subject: Re: [Talk] what's the best way to represent structured dates? > > Patrick, > > There was quite a bit of discussion of this in our Wednesday > afternoon meetings at Berkeley that I'm sure Aron will > recall. I don't think date fields go back far enough in time, > so I'd recommend using integers or floating point numbers and > providing the math in search. It is also important to make > sure that false precision isn't introduced. If something is > dated 1900, the system shouldn't enter that as January 1, > 1900 unless the uncertainty values can be used to > differentiate and the display date can be simply 1900. > > Susan > > > On 08/24/2011 09:16 AM, Patrick Schmitz wrote: > > Really? I am a little surprised that in all the design > discussions for > > structured dates this did not come up. > > Anyway, I will say that we do not have the time to build > some system > > that allows for a lot of custom behavior, or elaborate > heuristics here > > (i.e., trying to divine what is intended from a set of text > fields). I > > would strongly suggest that we either > > > > 1. > > require entry of some actual date fields upon which the > application > > can reason, > > or > > 2. > > we define a very simple set of rules for building such > date values > > from some of the existing text fields. By very simple, > I mean that > > we only consider fields like year, month, day, and that > the rules > > produce two scalar dates (early and late), which either define a > > range, or define a point (if they are equal). AIUI, a structured > > date /cannot/ model the idea of "some time/any time after (or > > before) a date". > > > > One further question is the semantic of which date (if any) will be > > used for ordering. > > If and when advanced search supports date ranges (which > also implies > > support for /before/ and /after/ logic in search), we can > handle the > > interval logic associated with the structured dates, but only if we > > have a pair of scalar date values. > > Down the road, I do think it would be interesting to implement some > > fuzzier logic on date search, to better support "circa" > logic and the > > uncertainty values. However, I suspect that it will take longer to > > arrive at consensus on these semantics, and would defer that > > discussion for now. > > Thanks - Patrick > > > > > -------------------------------------------------------------- > ---------- > > *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu] > > *Sent:* Wednesday, August 24, 2011 9:01 AM > > *To:* Patrick Schmitz > > *Cc:* 'Christopher Pott'; 'Aron Roberts'; > talk@lists.collectionspace.org > > *Subject:* Re: [Talk] what's the best way to represent > structured dates? > > > > Hi Patrick, I think it's fair to say there's no > consensus because > > there's been no discussion :-) It's possible that each > customer will > > want to use structured dates differently, which will > make it hard to > > add any intelligence or even predictability to things > like search. > > However, I also think that if a significant number of deploying > > institutions can come to some consensus on best ways to use and > > search structured dates, then many institutions will > choose to adopt > > the same best practices. > > > > Thanks, > > Chris > > > > On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote: > > > >> Aron was out yesterday, and Rick is out for a few > days. They have > >> done a lot of the structured dates work to date. > >> The last I heard, Aron was hoping to get consensus > from the users > >> about how we map all the text fields to actual date > fields that we > >> can reason on (for search, ordering, etc). To date, > there seems to > >> be no such consensus, which will preclude search, > ordering, etc. > >> on structured dates. I would think that this would > make structured > >> dates quite a bit less useful... > >> Perhaps (I hope) my understanding is out of date, and > we just need > >> to implement the logic. Perhaps Aron can jump in this > AM, when he > >> gets in. > >> Patrick > >> > >> > -------------------------------------------------------------- > ---------- > >> *From:* talk-bounces@lists.collectionspace.org > >> <mailto:talk-bounces@lists.collectionspace.org> > >> [mailto:talk-bounces@lists.collectionspace.org] > *On Behalf Of > >> *Chris Hoffman > >> *Sent:* Tuesday, August 23, 2011 11:05 AM > >> *To:* Christopher Pott > >> *Cc:* talk@lists.collectionspace.org > >> <mailto:talk@lists.collectionspace.org> > >> *Subject:* Re: [Talk] what's the best way to represent > >> structured dates? > >> > >> Hi Chris, > >> > >> Thanks for asking these questions. We've been wondering the > >> same things. Have you changed any existing calendar date > >> fields to structured dates or added new structured date > >> fields? I think this is all quite new, and > together we might > >> be able to come up with some best practices > (related to data > >> migration and data entry). > >> > >> I suspect that search and advanced search of > structured date > >> fields is going to be an issue for those of us deploying > >> CollectionSpace. Our users will want CSpace to be > smart about > >> structured dates, interpreting qualifiers, and so on, but I > >> bet that we won't see much as much as we would like in the > >> first version of advanced search. I'm always happy to be > >> proved wrong! :-) > >> > >> Chris > >> > >> On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote: > >> > >>> Hi, > >>> > >>> I'm looking at the structured date schema and the > best way to > >>> represent dates migrated from our old system. Any > information > >>> about the following would be much appreciated.. > >>> > >>> 1. I see some different/overlapping possibilities for > >>> recording an object creation date. For example, > to represent > >>> "1718" (display date) one could either store: > >>> > >>> a) a single date, providing year only (1718) > >>> > >>> b) an earliest date (01.01.1718) with a latest date > >>> (31.12.1718) > >>> > >>> c) a single date (01.01.1718) with a qualifier of > + 364 days > >>> > >>> Is there any difference in the meaning of these three (and > >>> would they all be treated equally by services such as > >>> advanced search)? Our migration data is currently > in format > >>> 'b' but the other formats are sometimes > 'friendlier' for new > >>> input. > >>> > >>> 2. How are date "certainties" and "qualifiers" used by > >>> advanced search? > >>> > >>> I've specified "1718", as a creation date with a > certainty of > >>> "circa". Will a search for objects created during > 1717 result > >>> in a hit? > >>> > >>> Or is it necessary to use qualifiers (eg. 1718 > +/- 2 years) > >>> to define exact limits for this type of searching? > >>> > >>> 3. Is the Date schema locally extendable? (We > need a field to > >>> hold an English translation of the display text) > >>> > >>> 4. Will Advanced search by creation date also search the > >>> display date or will it just use the single/earliest and > >>> latest date groups? > >>> > >>> Thanks, > >>> > >>> Chris > >>> > >>> _______________________________________________ > >>> Talk mailing list > >>> Talk@lists.collectionspace.org > >>> <mailto:Talk@lists.collectionspace.org> > >>> > >>> > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collect > >>> ionspace.org > >> > > > > > > > > _______________________________________________ > > Talk mailing list > > Talk@lists.collectionspace.org > > > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectio > > nspace.org > > > _______________________________________________ > Talk mailing list > Talk@lists.collectionspace.org > http://lists.collectionspace.org/mailman/listinfo/talk_lists.c > ollectionspace.org >
SS
Susan Stone
Wed, Aug 24, 2011 5:29 PM

Patrick,

Obviously 4713 BC is not early enough. Michael is your source for those
really early dates, but I would hope that since we also have a date
period field in the structured date, that could be used for those.

Thanks for looking at the custom type. I think it is probably the way to
go (especially if you don't have to go the Julian date route!).

Susan

On 08/24/2011 10:23 AM, Patrick Schmitz wrote:

Thanks, Susan.

PostgreSQL date supports the date range 4713 BC to 5874897 AD. Personally, I
wish they had balanced past dates better, but such is life.

MySQL is more restrictive, and only supports 1000 AD to 9999 AD. Sigh.

It appears that we will have to implement a custom data type for our dates.
A Java Gregorian calendar  (which can be converted to a signed long, and
stored in the DB as such) can represent dates back to almost 300 Million
years BCE (earliest year is 292269055 BCE). Would this be enough range? If
not, we'd have to do something ever funkier.

Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf Of
Susan Stone
Sent: Wednesday, August 24, 2011 9:34 AM
To: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent structured dates?

Patrick,

There was quite a bit of discussion of this in our Wednesday
afternoon meetings at Berkeley that I'm sure Aron will
recall. I don't think date fields go back far enough in time,
so I'd recommend using integers or floating point numbers and
providing the math in search. It is also important to make
sure that false precision isn't introduced. If something is
dated 1900, the system shouldn't enter that as January 1,
1900 unless the uncertainty values can be used to
differentiate and the display date can be simply 1900.

Susan

On 08/24/2011 09:16 AM, Patrick Schmitz wrote:

Really? I am a little surprised that in all the design

discussions for

structured dates this did not come up.
Anyway, I will say that we do not have the time to build

some system

that allows for a lot of custom behavior, or elaborate

heuristics here

(i.e., trying to divine what is intended from a set of text

fields). I

would strongly suggest that we either

  1. require entry of some actual date fields upon which the

application

  can reason,
  or
  1. we define a very simple set of rules for building such

date values

  from some of the existing text fields. By very simple,

I mean that

  we only consider fields like year, month, day, and that

the rules

  produce two scalar dates (early and late), which either define a
  range, or define a point (if they are equal). AIUI, a structured
  date /cannot/ model the idea of "some time/any time after (or
  before) a date".

One further question is the semantic of which date (if any) will be
used for ordering.
If and when advanced search supports date ranges (which

also implies

support for /before/ and /after/ logic in search), we can

handle the

interval logic associated with the structured dates, but only if we
have a pair of scalar date values.
Down the road, I do think it would be interesting to implement some
fuzzier logic on date search, to better support "circa"

logic and the

uncertainty values. However, I suspect that it will take longer to
arrive at consensus on these semantics, and would defer that
discussion for now.
Thanks - Patrick



  *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
  *Sent:* Wednesday, August 24, 2011 9:01 AM
  *To:* Patrick Schmitz
  *Cc:* 'Christopher Pott'; 'Aron Roberts';
  *Subject:* Re: [Talk] what's the best way to represent

structured dates?

  Hi Patrick, I think it's fair to say there's no

consensus because

  there's been no discussion :-) It's possible that each

customer will

  want to use structured dates differently, which will

make it hard to

  add any intelligence or even predictability to things

like search.

  However, I also think that if a significant number of deploying
  institutions can come to some consensus on best ways to use and
  search structured dates, then many institutions will

choose to adopt

  the same best practices.

  Thanks,
  Chris

  On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote:
  Aron was out yesterday, and Rick is out for a few

days. They have

  done a lot of the structured dates work to date.
  The last I heard, Aron was hoping to get consensus

from the users

  about how we map all the text fields to actual date

fields that we

  can reason on (for search, ordering, etc). To date,

there seems to

  be no such consensus, which will preclude search,

ordering, etc.

  on structured dates. I would think that this would

make structured

  dates quite a bit less useful...
  Perhaps (I hope) my understanding is out of date, and

we just need

  to implement the logic. Perhaps Aron can jump in this

AM, when he

  gets in.
  Patrick


      *From:* talk-bounces@lists.collectionspace.org
      <mailto:talk-bounces@lists.collectionspace.org>
      [mailto:talk-bounces@lists.collectionspace.org]

*On Behalf Of

      *Chris Hoffman
      *Sent:* Tuesday, August 23, 2011 11:05 AM
      *To:* Christopher Pott
      *Cc:* talk@lists.collectionspace.org
      <mailto:talk@lists.collectionspace.org>
      *Subject:* Re: [Talk] what's the best way to represent
      structured dates?

      Hi Chris,

      Thanks for asking these questions. We've been wondering the
      same things. Have you changed any existing calendar date
      fields to structured dates or added new structured date
      fields? I think this is all quite new, and

together we might

      be able to come up with some best practices

(related to data

      migration and data entry).

      I suspect that search and advanced search of

structured date

      fields is going to be an issue for those of us deploying
      CollectionSpace. Our users will want CSpace to be

smart about

      structured dates, interpreting qualifiers, and so on, but I
      bet that we won't see much as much as we would like in the
      first version of advanced search. I'm always happy to be
      proved wrong! :-)

      Chris

      On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote:
      Hi,

      I'm looking at the structured date schema and the

best way to

      represent dates migrated from our old system. Any

information

      about the following would be much appreciated..

      1. I see some different/overlapping possibilities for
      recording an object creation date. For example,

to represent

      "1718" (display date) one could either store:

      a) a single date, providing year only (1718)

      b) an earliest date (01.01.1718) with a latest date

(31.12.1718)

      c) a single date (01.01.1718) with a qualifier of
  • 364 days
      Is there any difference in the meaning of these three (and
      would they all be treated equally by services such as
      advanced search)? Our migration data is currently

in format

      'b' but the other formats are sometimes

'friendlier' for new

      input.

      2. How are date "certainties" and "qualifiers" used by
      advanced search?

      I've specified "1718", as a creation date with a

certainty of

      "circa". Will a search for objects created during

1717 result

      in a hit?

      Or is it necessary to use qualifiers (eg. 1718

+/- 2 years)

      to define exact limits for this type of searching?

      3. Is the Date schema locally extendable? (We

need a field to

      hold an English translation of the display text)

      4. Will Advanced search by creation date also search the
      display date or will it just use the single/earliest and
      latest date groups?

      Thanks,

      Chris

      _______________________________________________
      Talk mailing list
      Talk@lists.collectionspace.org
      <mailto:Talk@lists.collectionspace.org>

ionspace.org


Talk mailing list
Talk@lists.collectionspace.org

nspace.org

Patrick, Obviously 4713 BC is not early enough. Michael is your source for those really early dates, but I would hope that since we also have a date period field in the structured date, that could be used for those. Thanks for looking at the custom type. I think it is probably the way to go (especially if you don't have to go the Julian date route!). Susan On 08/24/2011 10:23 AM, Patrick Schmitz wrote: > Thanks, Susan. > > PostgreSQL date supports the date range 4713 BC to 5874897 AD. Personally, I > wish they had balanced past dates better, but such is life. > > MySQL is more restrictive, and only supports 1000 AD to 9999 AD. Sigh. > > It appears that we will have to implement a custom data type for our dates. > A Java Gregorian calendar (which can be converted to a signed long, and > stored in the DB as such) can represent dates back to almost 300 Million > years BCE (earliest year is 292269055 BCE). Would this be enough range? If > not, we'd have to do something ever funkier. > > Patrick > >> -----Original Message----- >> From: talk-bounces@lists.collectionspace.org >> [mailto:talk-bounces@lists.collectionspace.org] On Behalf Of >> Susan Stone >> Sent: Wednesday, August 24, 2011 9:34 AM >> To: talk@lists.collectionspace.org >> Subject: Re: [Talk] what's the best way to represent structured dates? >> >> Patrick, >> >> There was quite a bit of discussion of this in our Wednesday >> afternoon meetings at Berkeley that I'm sure Aron will >> recall. I don't think date fields go back far enough in time, >> so I'd recommend using integers or floating point numbers and >> providing the math in search. It is also important to make >> sure that false precision isn't introduced. If something is >> dated 1900, the system shouldn't enter that as January 1, >> 1900 unless the uncertainty values can be used to >> differentiate and the display date can be simply 1900. >> >> Susan >> >> >> On 08/24/2011 09:16 AM, Patrick Schmitz wrote: >>> Really? I am a little surprised that in all the design >> discussions for >>> structured dates this did not come up. >>> Anyway, I will say that we do not have the time to build >> some system >>> that allows for a lot of custom behavior, or elaborate >> heuristics here >>> (i.e., trying to divine what is intended from a set of text >> fields). I >>> would strongly suggest that we either >>> >>> 1. >>> require entry of some actual date fields upon which the >> application >>> can reason, >>> or >>> 2. >>> we define a very simple set of rules for building such >> date values >>> from some of the existing text fields. By very simple, >> I mean that >>> we only consider fields like year, month, day, and that >> the rules >>> produce two scalar dates (early and late), which either define a >>> range, or define a point (if they are equal). AIUI, a structured >>> date /cannot/ model the idea of "some time/any time after (or >>> before) a date". >>> >>> One further question is the semantic of which date (if any) will be >>> used for ordering. >>> If and when advanced search supports date ranges (which >> also implies >>> support for /before/ and /after/ logic in search), we can >> handle the >>> interval logic associated with the structured dates, but only if we >>> have a pair of scalar date values. >>> Down the road, I do think it would be interesting to implement some >>> fuzzier logic on date search, to better support "circa" >> logic and the >>> uncertainty values. However, I suspect that it will take longer to >>> arrive at consensus on these semantics, and would defer that >>> discussion for now. >>> Thanks - Patrick >>> >>> >> -------------------------------------------------------------- >> ---------- >>> *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu] >>> *Sent:* Wednesday, August 24, 2011 9:01 AM >>> *To:* Patrick Schmitz >>> *Cc:* 'Christopher Pott'; 'Aron Roberts'; >> talk@lists.collectionspace.org >>> *Subject:* Re: [Talk] what's the best way to represent >> structured dates? >>> >>> Hi Patrick, I think it's fair to say there's no >> consensus because >>> there's been no discussion :-) It's possible that each >> customer will >>> want to use structured dates differently, which will >> make it hard to >>> add any intelligence or even predictability to things >> like search. >>> However, I also think that if a significant number of deploying >>> institutions can come to some consensus on best ways to use and >>> search structured dates, then many institutions will >> choose to adopt >>> the same best practices. >>> >>> Thanks, >>> Chris >>> >>> On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote: >>> >>>> Aron was out yesterday, and Rick is out for a few >> days. They have >>>> done a lot of the structured dates work to date. >>>> The last I heard, Aron was hoping to get consensus >> from the users >>>> about how we map all the text fields to actual date >> fields that we >>>> can reason on (for search, ordering, etc). To date, >> there seems to >>>> be no such consensus, which will preclude search, >> ordering, etc. >>>> on structured dates. I would think that this would >> make structured >>>> dates quite a bit less useful... >>>> Perhaps (I hope) my understanding is out of date, and >> we just need >>>> to implement the logic. Perhaps Aron can jump in this >> AM, when he >>>> gets in. >>>> Patrick >>>> >>>> >> -------------------------------------------------------------- >> ---------- >>>> *From:* talk-bounces@lists.collectionspace.org >>>> <mailto:talk-bounces@lists.collectionspace.org> >>>> [mailto:talk-bounces@lists.collectionspace.org] >> *On Behalf Of >>>> *Chris Hoffman >>>> *Sent:* Tuesday, August 23, 2011 11:05 AM >>>> *To:* Christopher Pott >>>> *Cc:* talk@lists.collectionspace.org >>>> <mailto:talk@lists.collectionspace.org> >>>> *Subject:* Re: [Talk] what's the best way to represent >>>> structured dates? >>>> >>>> Hi Chris, >>>> >>>> Thanks for asking these questions. We've been wondering the >>>> same things. Have you changed any existing calendar date >>>> fields to structured dates or added new structured date >>>> fields? I think this is all quite new, and >> together we might >>>> be able to come up with some best practices >> (related to data >>>> migration and data entry). >>>> >>>> I suspect that search and advanced search of >> structured date >>>> fields is going to be an issue for those of us deploying >>>> CollectionSpace. Our users will want CSpace to be >> smart about >>>> structured dates, interpreting qualifiers, and so on, but I >>>> bet that we won't see much as much as we would like in the >>>> first version of advanced search. I'm always happy to be >>>> proved wrong! :-) >>>> >>>> Chris >>>> >>>> On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm looking at the structured date schema and the >> best way to >>>>> represent dates migrated from our old system. Any >> information >>>>> about the following would be much appreciated.. >>>>> >>>>> 1. I see some different/overlapping possibilities for >>>>> recording an object creation date. For example, >> to represent >>>>> "1718" (display date) one could either store: >>>>> >>>>> a) a single date, providing year only (1718) >>>>> >>>>> b) an earliest date (01.01.1718) with a latest date >>>>> (31.12.1718) >>>>> >>>>> c) a single date (01.01.1718) with a qualifier of >> + 364 days >>>>> >>>>> Is there any difference in the meaning of these three (and >>>>> would they all be treated equally by services such as >>>>> advanced search)? Our migration data is currently >> in format >>>>> 'b' but the other formats are sometimes >> 'friendlier' for new >>>>> input. >>>>> >>>>> 2. How are date "certainties" and "qualifiers" used by >>>>> advanced search? >>>>> >>>>> I've specified "1718", as a creation date with a >> certainty of >>>>> "circa". Will a search for objects created during >> 1717 result >>>>> in a hit? >>>>> >>>>> Or is it necessary to use qualifiers (eg. 1718 >> +/- 2 years) >>>>> to define exact limits for this type of searching? >>>>> >>>>> 3. Is the Date schema locally extendable? (We >> need a field to >>>>> hold an English translation of the display text) >>>>> >>>>> 4. Will Advanced search by creation date also search the >>>>> display date or will it just use the single/earliest and >>>>> latest date groups? >>>>> >>>>> Thanks, >>>>> >>>>> Chris >>>>> >>>>> _______________________________________________ >>>>> Talk mailing list >>>>> Talk@lists.collectionspace.org >>>>> <mailto:Talk@lists.collectionspace.org> >>>>> >>>>> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collect >>>>> ionspace.org >>>> >>> >>> >>> >>> _______________________________________________ >>> Talk mailing list >>> Talk@lists.collectionspace.org >>> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectio >>> nspace.org >> >> >> _______________________________________________ >> Talk mailing list >> Talk@lists.collectionspace.org >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c >> ollectionspace.org >> >
PS
Patrick Schmitz
Wed, Aug 24, 2011 6:05 PM

Maybe I am missing something, but is there a problem with the following:

Single date, expressed as "1910". We model this (in earliest and latest
scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910"

Date range expressed as earliest:"1910", latest:"1915". We model this as
esd: "Jan 1 1910" and lsd: "Dec 31 1915"

Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We model
this as esd: "Jan 1 1910" and lsd: "Aug 31 1910"

The esd and lsd scalar dates are used for search and ordering, not display.

Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf Of
Janice Eklund
Sent: Wednesday, August 24, 2011 10:17 AM
To: Susan Stone
Cc: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent structured dates?

I second Susan's concern about false precision.  At least 90%
of our dates are year only single dates or date ranges. If
you are considering fuzzy logic down the road, look at the
existing code carefully.  The fuzzy date machine, at least as
implemented at UCB for the HAVRC, does not handle ancient
dates properly.

Jan

On 8/24/11, Susan Stone sstone@socrates.berkeley.edu wrote:

Patrick,

There was quite a bit of discussion of this in our

Wednesday afternoon

meetings at Berkeley that I'm sure Aron will recall. I don't think
date fields go back far enough in time, so I'd recommend using
integers or floating point numbers and providing the math

in search.

It is also important to make sure that false precision isn't
introduced. If something is dated 1900, the system shouldn't enter
that as January 1, 1900 unless the uncertainty values can

be used to

differentiate and the display date can be simply 1900.

Susan

On 08/24/2011 09:16 AM, Patrick Schmitz wrote:

Really? I am a little surprised that in all the design discussions
for structured dates this did not come up.
Anyway, I will say that we do not have the time to build

some system

that allows for a lot of custom behavior, or elaborate heuristics
here (i.e., trying to divine what is intended from a set of text
fields). I would strongly suggest that we either

  1. require entry of some actual date fields upon which

the application

 can reason,
 or
  1. we define a very simple set of rules for building such

date values

 from some of the existing text fields. By very simple, 

I mean that

 we only consider fields like year, month, day, and 

that the rules

 produce two scalar dates (early and late), which 

either define a

 range, or define a point (if they are equal). AIUI, a 

structured

 date /cannot/ model the idea of "some time/any time after (or
 before) a date".

One further question is the semantic of which date (if

any) will be

used for ordering.
If and when advanced search supports date ranges (which

also implies

support for /before/ and /after/ logic in search), we can

handle the

interval logic associated with the structured dates, but

only if we

have a pair of scalar date values.
Down the road, I do think it would be interesting to

implement some

fuzzier logic on date search, to better support "circa"

logic and the

uncertainty values. However, I suspect that it will take longer to
arrive at consensus on these semantics, and would defer that
discussion for now.
Thanks - Patrick



 *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
 *Sent:* Wednesday, August 24, 2011 9:01 AM
 *To:* Patrick Schmitz
 *Cc:* 'Christopher Pott'; 'Aron Roberts'; 

talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured

dates?

 Hi Patrick, I think it's fair to say there's no 

consensus because

 there's been no discussion :-) It's possible that each 

customer will

 want to use structured dates differently, which will 

make it hard to

 add any intelligence or even predictability to things 

like search.

 However, I also think that if a significant number of deploying
 institutions can come to some consensus on best ways to use and
 search structured dates, then many institutions will 

choose to adopt

 the same best practices.

 Thanks,
 Chris

 On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote:
 Aron was out yesterday, and Rick is out for a few 

days. They have

 done a lot of the structured dates work to date.
 The last I heard, Aron was hoping to get consensus 

from the users

 about how we map all the text fields to actual date 

fields that we

 can reason on (for search, ordering, etc). To date, 

there seems to

 be no such consensus, which will preclude search, 

ordering, etc.

 on structured dates. I would think that this would 

make structured

 dates quite a bit less useful...
 Perhaps (I hope) my understanding is out of date, and 

we just need

 to implement the logic. Perhaps Aron can jump in this 

AM, when he

 gets in.
 Patrick


     *From:* talk-bounces@lists.collectionspace.org
     <mailto:talk-bounces@lists.collectionspace.org>
     [mailto:talk-bounces@lists.collectionspace.org] 

*On Behalf Of

     *Chris Hoffman
     *Sent:* Tuesday, August 23, 2011 11:05 AM
     *To:* Christopher Pott
     *Cc:* talk@lists.collectionspace.org
     <mailto:talk@lists.collectionspace.org>
     *Subject:* Re: [Talk] what's the best way to represent
     structured dates?

     Hi Chris,

     Thanks for asking these questions. We've been 

wondering the

     same things. Have you changed any existing calendar date
     fields to structured dates or added new structured date
     fields? I think this is all quite new, and 

together we might

     be able to come up with some best practices 

(related to data

     migration and data entry).

     I suspect that search and advanced search of 

structured date

     fields is going to be an issue for those of us deploying
     CollectionSpace. Our users will want CSpace to be 

smart about

     structured dates, interpreting qualifiers, and so 

on, but I

     bet that we won't see much as much as we would like in the
     first version of advanced search. I'm always happy to be
     proved wrong! :-)

     Chris

     On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote:
     Hi,

     I'm looking at the structured date schema and 

the best way to

     represent dates migrated from our old system. 

Any information

     about the following would be much appreciated..

     1. I see some different/overlapping possibilities for
     recording an object creation date. For example, 

to represent

     "1718" (display date) one could either store:

     a) a single date, providing year only (1718)

     b) an earliest date (01.01.1718) with a latest date 

(31.12.1718)

     c) a single date (01.01.1718) with a qualifier of + 364 

days

     Is there any difference in the meaning of these 

three (and

     would they all be treated equally by services such as
     advanced search)? Our migration data is 

currently in format

     'b' but the other formats are sometimes 

'friendlier' for new

     input.

     2. How are date "certainties" and "qualifiers" used by
     advanced search?

     I've specified "1718", as a creation date with a 

certainty of

     "circa". Will a search for objects created 

during 1717 result

     in a hit?

     Or is it necessary to use qualifiers (eg. 1718 

+/- 2 years)

     to define exact limits for this type of searching?

     3. Is the Date schema locally extendable? (We 

need a field to

     hold an English translation of the display text)

     4. Will Advanced search by creation date also search the
     display date or will it just use the single/earliest and
     latest date groups?

     Thanks,

     Chris

     _______________________________________________
     Talk mailing list
     Talk@lists.collectionspace.org
     <mailto:Talk@lists.collectionspace.org>

tionspace.org


Talk mailing list
Talk@lists.collectionspace.org

onspace.org


Talk mailing list
Talk@lists.collectionspace.org

nspace.org

Maybe I am missing something, but is there a problem with the following: Single date, expressed as "1910". We model this (in earliest and latest scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910" Date range expressed as earliest:"1910", latest:"1915". We model this as esd: "Jan 1 1910" and lsd: "Dec 31 1915" Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We model this as esd: "Jan 1 1910" and lsd: "Aug 31 1910" The esd and lsd scalar dates are used for search and ordering, not display. Patrick > -----Original Message----- > From: talk-bounces@lists.collectionspace.org > [mailto:talk-bounces@lists.collectionspace.org] On Behalf Of > Janice Eklund > Sent: Wednesday, August 24, 2011 10:17 AM > To: Susan Stone > Cc: talk@lists.collectionspace.org > Subject: Re: [Talk] what's the best way to represent structured dates? > > I second Susan's concern about false precision. At least 90% > of our dates are year only single dates or date ranges. If > you are considering fuzzy logic down the road, look at the > existing code carefully. The fuzzy date machine, at least as > implemented at UCB for the HAVRC, does not handle ancient > dates properly. > > Jan > > On 8/24/11, Susan Stone <sstone@socrates.berkeley.edu> wrote: > > Patrick, > > > > There was quite a bit of discussion of this in our > Wednesday afternoon > > meetings at Berkeley that I'm sure Aron will recall. I don't think > > date fields go back far enough in time, so I'd recommend using > > integers or floating point numbers and providing the math > in search. > > It is also important to make sure that false precision isn't > > introduced. If something is dated 1900, the system shouldn't enter > > that as January 1, 1900 unless the uncertainty values can > be used to > > differentiate and the display date can be simply 1900. > > > > Susan > > > > > > On 08/24/2011 09:16 AM, Patrick Schmitz wrote: > >> Really? I am a little surprised that in all the design discussions > >> for structured dates this did not come up. > >> Anyway, I will say that we do not have the time to build > some system > >> that allows for a lot of custom behavior, or elaborate heuristics > >> here (i.e., trying to divine what is intended from a set of text > >> fields). I would strongly suggest that we either > >> > >> 1. > >> require entry of some actual date fields upon which > the application > >> can reason, > >> or > >> 2. > >> we define a very simple set of rules for building such > date values > >> from some of the existing text fields. By very simple, > I mean that > >> we only consider fields like year, month, day, and > that the rules > >> produce two scalar dates (early and late), which > either define a > >> range, or define a point (if they are equal). AIUI, a > structured > >> date /cannot/ model the idea of "some time/any time after (or > >> before) a date". > >> > >> One further question is the semantic of which date (if > any) will be > >> used for ordering. > >> If and when advanced search supports date ranges (which > also implies > >> support for /before/ and /after/ logic in search), we can > handle the > >> interval logic associated with the structured dates, but > only if we > >> have a pair of scalar date values. > >> Down the road, I do think it would be interesting to > implement some > >> fuzzier logic on date search, to better support "circa" > logic and the > >> uncertainty values. However, I suspect that it will take longer to > >> arrive at consensus on these semantics, and would defer that > >> discussion for now. > >> Thanks - Patrick > >> > >> > >> > -------------------------------------------------------------- > ---------- > >> *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu] > >> *Sent:* Wednesday, August 24, 2011 9:01 AM > >> *To:* Patrick Schmitz > >> *Cc:* 'Christopher Pott'; 'Aron Roberts'; > >> talk@lists.collectionspace.org > >> *Subject:* Re: [Talk] what's the best way to represent > structured > >> dates? > >> > >> Hi Patrick, I think it's fair to say there's no > consensus because > >> there's been no discussion :-) It's possible that each > customer will > >> want to use structured dates differently, which will > make it hard to > >> add any intelligence or even predictability to things > like search. > >> However, I also think that if a significant number of deploying > >> institutions can come to some consensus on best ways to use and > >> search structured dates, then many institutions will > choose to adopt > >> the same best practices. > >> > >> Thanks, > >> Chris > >> > >> On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote: > >> > >>> Aron was out yesterday, and Rick is out for a few > days. They have > >>> done a lot of the structured dates work to date. > >>> The last I heard, Aron was hoping to get consensus > from the users > >>> about how we map all the text fields to actual date > fields that we > >>> can reason on (for search, ordering, etc). To date, > there seems to > >>> be no such consensus, which will preclude search, > ordering, etc. > >>> on structured dates. I would think that this would > make structured > >>> dates quite a bit less useful... > >>> Perhaps (I hope) my understanding is out of date, and > we just need > >>> to implement the logic. Perhaps Aron can jump in this > AM, when he > >>> gets in. > >>> Patrick > >>> > >>> > >>> > -------------------------------------------------------------- > ---------- > >>> *From:* talk-bounces@lists.collectionspace.org > >>> <mailto:talk-bounces@lists.collectionspace.org> > >>> [mailto:talk-bounces@lists.collectionspace.org] > *On Behalf Of > >>> *Chris Hoffman > >>> *Sent:* Tuesday, August 23, 2011 11:05 AM > >>> *To:* Christopher Pott > >>> *Cc:* talk@lists.collectionspace.org > >>> <mailto:talk@lists.collectionspace.org> > >>> *Subject:* Re: [Talk] what's the best way to represent > >>> structured dates? > >>> > >>> Hi Chris, > >>> > >>> Thanks for asking these questions. We've been > wondering the > >>> same things. Have you changed any existing calendar date > >>> fields to structured dates or added new structured date > >>> fields? I think this is all quite new, and > together we might > >>> be able to come up with some best practices > (related to data > >>> migration and data entry). > >>> > >>> I suspect that search and advanced search of > structured date > >>> fields is going to be an issue for those of us deploying > >>> CollectionSpace. Our users will want CSpace to be > smart about > >>> structured dates, interpreting qualifiers, and so > on, but I > >>> bet that we won't see much as much as we would like in the > >>> first version of advanced search. I'm always happy to be > >>> proved wrong! :-) > >>> > >>> Chris > >>> > >>> On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote: > >>> > >>>> Hi, > >>>> > >>>> I'm looking at the structured date schema and > the best way to > >>>> represent dates migrated from our old system. > Any information > >>>> about the following would be much appreciated.. > >>>> > >>>> 1. I see some different/overlapping possibilities for > >>>> recording an object creation date. For example, > to represent > >>>> "1718" (display date) one could either store: > >>>> > >>>> a) a single date, providing year only (1718) > >>>> > >>>> b) an earliest date (01.01.1718) with a latest date > >>>> (31.12.1718) > >>>> > >>>> c) a single date (01.01.1718) with a qualifier of + 364 > >>>> days > >>>> > >>>> Is there any difference in the meaning of these > three (and > >>>> would they all be treated equally by services such as > >>>> advanced search)? Our migration data is > currently in format > >>>> 'b' but the other formats are sometimes > 'friendlier' for new > >>>> input. > >>>> > >>>> 2. How are date "certainties" and "qualifiers" used by > >>>> advanced search? > >>>> > >>>> I've specified "1718", as a creation date with a > certainty of > >>>> "circa". Will a search for objects created > during 1717 result > >>>> in a hit? > >>>> > >>>> Or is it necessary to use qualifiers (eg. 1718 > +/- 2 years) > >>>> to define exact limits for this type of searching? > >>>> > >>>> 3. Is the Date schema locally extendable? (We > need a field to > >>>> hold an English translation of the display text) > >>>> > >>>> 4. Will Advanced search by creation date also search the > >>>> display date or will it just use the single/earliest and > >>>> latest date groups? > >>>> > >>>> Thanks, > >>>> > >>>> Chris > >>>> > >>>> _______________________________________________ > >>>> Talk mailing list > >>>> Talk@lists.collectionspace.org > >>>> <mailto:Talk@lists.collectionspace.org> > >>>> > >>>> > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collec > >>>> tionspace.org > >>> > >> > >> > >> > >> _______________________________________________ > >> Talk mailing list > >> Talk@lists.collectionspace.org > >> > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collecti > >> onspace.org > > > > > > _______________________________________________ > > Talk mailing list > > Talk@lists.collectionspace.org > > > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectio > > nspace.org > > > > _______________________________________________ > Talk mailing list > Talk@lists.collectionspace.org > http://lists.collectionspace.org/mailman/listinfo/talk_lists.c > ollectionspace.org >
JE
Janice Eklund
Wed, Aug 24, 2011 6:16 PM

Patrick,

I see no problem with the scalar date scenarios listed below.  Problem
is with the way things work now is that the date defaults to day one
of the current month regardless of the year date entered.  So if I
just enter 1910 in the Associated date field of the catalog record, it
returns 1910-08-01.  I realize this is the calendar date widget at
work, but I just wanted to make sure this wouldn't do the same thing
in the structured date.

Jan

On 8/24/11, Patrick Schmitz pschmitz@berkeley.edu wrote:

Maybe I am missing something, but is there a problem with the following:

Single date, expressed as "1910". We model this (in earliest and latest
scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910"

Date range expressed as earliest:"1910", latest:"1915". We model this as
esd: "Jan 1 1910" and lsd: "Dec 31 1915"

Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We model
this as esd: "Jan 1 1910" and lsd: "Aug 31 1910"

The esd and lsd scalar dates are used for search and ordering, not display.

Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf Of
Janice Eklund
Sent: Wednesday, August 24, 2011 10:17 AM
To: Susan Stone
Cc: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent structured dates?

I second Susan's concern about false precision.  At least 90%
of our dates are year only single dates or date ranges. If
you are considering fuzzy logic down the road, look at the
existing code carefully.  The fuzzy date machine, at least as
implemented at UCB for the HAVRC, does not handle ancient
dates properly.

Jan

On 8/24/11, Susan Stone sstone@socrates.berkeley.edu wrote:

Patrick,

There was quite a bit of discussion of this in our

Wednesday afternoon

meetings at Berkeley that I'm sure Aron will recall. I don't think
date fields go back far enough in time, so I'd recommend using
integers or floating point numbers and providing the math

in search.

It is also important to make sure that false precision isn't
introduced. If something is dated 1900, the system shouldn't enter
that as January 1, 1900 unless the uncertainty values can

be used to

differentiate and the display date can be simply 1900.

Susan

On 08/24/2011 09:16 AM, Patrick Schmitz wrote:

Really? I am a little surprised that in all the design discussions
for structured dates this did not come up.
Anyway, I will say that we do not have the time to build

some system

that allows for a lot of custom behavior, or elaborate heuristics
here (i.e., trying to divine what is intended from a set of text
fields). I would strongly suggest that we either

  1. require entry of some actual date fields upon which

the application

 can reason,
 or
  1. we define a very simple set of rules for building such

date values

 from some of the existing text fields. By very simple,

I mean that

 we only consider fields like year, month, day, and

that the rules

 produce two scalar dates (early and late), which

either define a

 range, or define a point (if they are equal). AIUI, a

structured

 date /cannot/ model the idea of "some time/any time after (or
 before) a date".

One further question is the semantic of which date (if

any) will be

used for ordering.
If and when advanced search supports date ranges (which

also implies

support for /before/ and /after/ logic in search), we can

handle the

interval logic associated with the structured dates, but

only if we

have a pair of scalar date values.
Down the road, I do think it would be interesting to

implement some

fuzzier logic on date search, to better support "circa"

logic and the

uncertainty values. However, I suspect that it will take longer to
arrive at consensus on these semantics, and would defer that
discussion for now.
Thanks - Patrick



 *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
 *Sent:* Wednesday, August 24, 2011 9:01 AM
 *To:* Patrick Schmitz
 *Cc:* 'Christopher Pott'; 'Aron Roberts';

talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured

dates?

 Hi Patrick, I think it's fair to say there's no

consensus because

 there's been no discussion :-) It's possible that each

customer will

 want to use structured dates differently, which will

make it hard to

 add any intelligence or even predictability to things

like search.

 However, I also think that if a significant number of deploying
 institutions can come to some consensus on best ways to use and
 search structured dates, then many institutions will

choose to adopt

 the same best practices.

 Thanks,
 Chris

 On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote:
 Aron was out yesterday, and Rick is out for a few

days. They have

 done a lot of the structured dates work to date.
 The last I heard, Aron was hoping to get consensus

from the users

 about how we map all the text fields to actual date

fields that we

 can reason on (for search, ordering, etc). To date,

there seems to

 be no such consensus, which will preclude search,

ordering, etc.

 on structured dates. I would think that this would

make structured

 dates quite a bit less useful...
 Perhaps (I hope) my understanding is out of date, and

we just need

 to implement the logic. Perhaps Aron can jump in this

AM, when he

 gets in.
 Patrick


     *From:* talk-bounces@lists.collectionspace.org
     <mailto:talk-bounces@lists.collectionspace.org>
     [mailto:talk-bounces@lists.collectionspace.org]

*On Behalf Of

     *Chris Hoffman
     *Sent:* Tuesday, August 23, 2011 11:05 AM
     *To:* Christopher Pott
     *Cc:* talk@lists.collectionspace.org
     <mailto:talk@lists.collectionspace.org>
     *Subject:* Re: [Talk] what's the best way to represent
     structured dates?

     Hi Chris,

     Thanks for asking these questions. We've been

wondering the

     same things. Have you changed any existing calendar date
     fields to structured dates or added new structured date
     fields? I think this is all quite new, and

together we might

     be able to come up with some best practices

(related to data

     migration and data entry).

     I suspect that search and advanced search of

structured date

     fields is going to be an issue for those of us deploying
     CollectionSpace. Our users will want CSpace to be

smart about

     structured dates, interpreting qualifiers, and so

on, but I

     bet that we won't see much as much as we would like in the
     first version of advanced search. I'm always happy to be
     proved wrong! :-)

     Chris

     On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote:
     Hi,

     I'm looking at the structured date schema and

the best way to

     represent dates migrated from our old system.

Any information

     about the following would be much appreciated..

     1. I see some different/overlapping possibilities for
     recording an object creation date. For example,

to represent

     "1718" (display date) one could either store:

     a) a single date, providing year only (1718)

     b) an earliest date (01.01.1718) with a latest date

(31.12.1718)

     c) a single date (01.01.1718) with a qualifier of + 364

days

     Is there any difference in the meaning of these

three (and

     would they all be treated equally by services such as
     advanced search)? Our migration data is

currently in format

     'b' but the other formats are sometimes

'friendlier' for new

     input.

     2. How are date "certainties" and "qualifiers" used by
     advanced search?

     I've specified "1718", as a creation date with a

certainty of

     "circa". Will a search for objects created

during 1717 result

     in a hit?

     Or is it necessary to use qualifiers (eg. 1718

+/- 2 years)

     to define exact limits for this type of searching?

     3. Is the Date schema locally extendable? (We

need a field to

     hold an English translation of the display text)

     4. Will Advanced search by creation date also search the
     display date or will it just use the single/earliest and
     latest date groups?

     Thanks,

     Chris

     _______________________________________________
     Talk mailing list
     Talk@lists.collectionspace.org
     <mailto:Talk@lists.collectionspace.org>

tionspace.org


Talk mailing list
Talk@lists.collectionspace.org

onspace.org


Talk mailing list
Talk@lists.collectionspace.org

nspace.org

Patrick, I see no problem with the scalar date scenarios listed below. Problem is with the way things work now is that the date defaults to day one of the current month regardless of the year date entered. So if I just enter 1910 in the Associated date field of the catalog record, it returns 1910-08-01. I realize this is the calendar date widget at work, but I just wanted to make sure this wouldn't do the same thing in the structured date. Jan On 8/24/11, Patrick Schmitz <pschmitz@berkeley.edu> wrote: > Maybe I am missing something, but is there a problem with the following: > > Single date, expressed as "1910". We model this (in earliest and latest > scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910" > > Date range expressed as earliest:"1910", latest:"1915". We model this as > esd: "Jan 1 1910" and lsd: "Dec 31 1915" > > Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We model > this as esd: "Jan 1 1910" and lsd: "Aug 31 1910" > > The esd and lsd scalar dates are used for search and ordering, not display. > > Patrick > >> -----Original Message----- >> From: talk-bounces@lists.collectionspace.org >> [mailto:talk-bounces@lists.collectionspace.org] On Behalf Of >> Janice Eklund >> Sent: Wednesday, August 24, 2011 10:17 AM >> To: Susan Stone >> Cc: talk@lists.collectionspace.org >> Subject: Re: [Talk] what's the best way to represent structured dates? >> >> I second Susan's concern about false precision. At least 90% >> of our dates are year only single dates or date ranges. If >> you are considering fuzzy logic down the road, look at the >> existing code carefully. The fuzzy date machine, at least as >> implemented at UCB for the HAVRC, does not handle ancient >> dates properly. >> >> Jan >> >> On 8/24/11, Susan Stone <sstone@socrates.berkeley.edu> wrote: >> > Patrick, >> > >> > There was quite a bit of discussion of this in our >> Wednesday afternoon >> > meetings at Berkeley that I'm sure Aron will recall. I don't think >> > date fields go back far enough in time, so I'd recommend using >> > integers or floating point numbers and providing the math >> in search. >> > It is also important to make sure that false precision isn't >> > introduced. If something is dated 1900, the system shouldn't enter >> > that as January 1, 1900 unless the uncertainty values can >> be used to >> > differentiate and the display date can be simply 1900. >> > >> > Susan >> > >> > >> > On 08/24/2011 09:16 AM, Patrick Schmitz wrote: >> >> Really? I am a little surprised that in all the design discussions >> >> for structured dates this did not come up. >> >> Anyway, I will say that we do not have the time to build >> some system >> >> that allows for a lot of custom behavior, or elaborate heuristics >> >> here (i.e., trying to divine what is intended from a set of text >> >> fields). I would strongly suggest that we either >> >> >> >> 1. >> >> require entry of some actual date fields upon which >> the application >> >> can reason, >> >> or >> >> 2. >> >> we define a very simple set of rules for building such >> date values >> >> from some of the existing text fields. By very simple, >> I mean that >> >> we only consider fields like year, month, day, and >> that the rules >> >> produce two scalar dates (early and late), which >> either define a >> >> range, or define a point (if they are equal). AIUI, a >> structured >> >> date /cannot/ model the idea of "some time/any time after (or >> >> before) a date". >> >> >> >> One further question is the semantic of which date (if >> any) will be >> >> used for ordering. >> >> If and when advanced search supports date ranges (which >> also implies >> >> support for /before/ and /after/ logic in search), we can >> handle the >> >> interval logic associated with the structured dates, but >> only if we >> >> have a pair of scalar date values. >> >> Down the road, I do think it would be interesting to >> implement some >> >> fuzzier logic on date search, to better support "circa" >> logic and the >> >> uncertainty values. However, I suspect that it will take longer to >> >> arrive at consensus on these semantics, and would defer that >> >> discussion for now. >> >> Thanks - Patrick >> >> >> >> >> >> >> -------------------------------------------------------------- >> ---------- >> >> *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu] >> >> *Sent:* Wednesday, August 24, 2011 9:01 AM >> >> *To:* Patrick Schmitz >> >> *Cc:* 'Christopher Pott'; 'Aron Roberts'; >> >> talk@lists.collectionspace.org >> >> *Subject:* Re: [Talk] what's the best way to represent >> structured >> >> dates? >> >> >> >> Hi Patrick, I think it's fair to say there's no >> consensus because >> >> there's been no discussion :-) It's possible that each >> customer will >> >> want to use structured dates differently, which will >> make it hard to >> >> add any intelligence or even predictability to things >> like search. >> >> However, I also think that if a significant number of deploying >> >> institutions can come to some consensus on best ways to use and >> >> search structured dates, then many institutions will >> choose to adopt >> >> the same best practices. >> >> >> >> Thanks, >> >> Chris >> >> >> >> On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote: >> >> >> >>> Aron was out yesterday, and Rick is out for a few >> days. They have >> >>> done a lot of the structured dates work to date. >> >>> The last I heard, Aron was hoping to get consensus >> from the users >> >>> about how we map all the text fields to actual date >> fields that we >> >>> can reason on (for search, ordering, etc). To date, >> there seems to >> >>> be no such consensus, which will preclude search, >> ordering, etc. >> >>> on structured dates. I would think that this would >> make structured >> >>> dates quite a bit less useful... >> >>> Perhaps (I hope) my understanding is out of date, and >> we just need >> >>> to implement the logic. Perhaps Aron can jump in this >> AM, when he >> >>> gets in. >> >>> Patrick >> >>> >> >>> >> >>> >> -------------------------------------------------------------- >> ---------- >> >>> *From:* talk-bounces@lists.collectionspace.org >> >>> <mailto:talk-bounces@lists.collectionspace.org> >> >>> [mailto:talk-bounces@lists.collectionspace.org] >> *On Behalf Of >> >>> *Chris Hoffman >> >>> *Sent:* Tuesday, August 23, 2011 11:05 AM >> >>> *To:* Christopher Pott >> >>> *Cc:* talk@lists.collectionspace.org >> >>> <mailto:talk@lists.collectionspace.org> >> >>> *Subject:* Re: [Talk] what's the best way to represent >> >>> structured dates? >> >>> >> >>> Hi Chris, >> >>> >> >>> Thanks for asking these questions. We've been >> wondering the >> >>> same things. Have you changed any existing calendar date >> >>> fields to structured dates or added new structured date >> >>> fields? I think this is all quite new, and >> together we might >> >>> be able to come up with some best practices >> (related to data >> >>> migration and data entry). >> >>> >> >>> I suspect that search and advanced search of >> structured date >> >>> fields is going to be an issue for those of us deploying >> >>> CollectionSpace. Our users will want CSpace to be >> smart about >> >>> structured dates, interpreting qualifiers, and so >> on, but I >> >>> bet that we won't see much as much as we would like in the >> >>> first version of advanced search. I'm always happy to be >> >>> proved wrong! :-) >> >>> >> >>> Chris >> >>> >> >>> On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote: >> >>> >> >>>> Hi, >> >>>> >> >>>> I'm looking at the structured date schema and >> the best way to >> >>>> represent dates migrated from our old system. >> Any information >> >>>> about the following would be much appreciated.. >> >>>> >> >>>> 1. I see some different/overlapping possibilities for >> >>>> recording an object creation date. For example, >> to represent >> >>>> "1718" (display date) one could either store: >> >>>> >> >>>> a) a single date, providing year only (1718) >> >>>> >> >>>> b) an earliest date (01.01.1718) with a latest date >> >>>> (31.12.1718) >> >>>> >> >>>> c) a single date (01.01.1718) with a qualifier of + 364 >> >>>> days >> >>>> >> >>>> Is there any difference in the meaning of these >> three (and >> >>>> would they all be treated equally by services such as >> >>>> advanced search)? Our migration data is >> currently in format >> >>>> 'b' but the other formats are sometimes >> 'friendlier' for new >> >>>> input. >> >>>> >> >>>> 2. How are date "certainties" and "qualifiers" used by >> >>>> advanced search? >> >>>> >> >>>> I've specified "1718", as a creation date with a >> certainty of >> >>>> "circa". Will a search for objects created >> during 1717 result >> >>>> in a hit? >> >>>> >> >>>> Or is it necessary to use qualifiers (eg. 1718 >> +/- 2 years) >> >>>> to define exact limits for this type of searching? >> >>>> >> >>>> 3. Is the Date schema locally extendable? (We >> need a field to >> >>>> hold an English translation of the display text) >> >>>> >> >>>> 4. Will Advanced search by creation date also search the >> >>>> display date or will it just use the single/earliest and >> >>>> latest date groups? >> >>>> >> >>>> Thanks, >> >>>> >> >>>> Chris >> >>>> >> >>>> _______________________________________________ >> >>>> Talk mailing list >> >>>> Talk@lists.collectionspace.org >> >>>> <mailto:Talk@lists.collectionspace.org> >> >>>> >> >>>> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collec >> >>>> tionspace.org >> >>> >> >> >> >> >> >> >> >> _______________________________________________ >> >> Talk mailing list >> >> Talk@lists.collectionspace.org >> >> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collecti >> >> onspace.org >> > >> > >> > _______________________________________________ >> > Talk mailing list >> > Talk@lists.collectionspace.org >> > >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectio >> > nspace.org >> > >> >> _______________________________________________ >> Talk mailing list >> Talk@lists.collectionspace.org >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c >> ollectionspace.org >> > >
AR
Aron Roberts
Wed, Aug 24, 2011 6:38 PM

On Wed, Aug 24, 2011 at 11:16 AM, Janice Eklund
janice.l.eklund@gmail.com wrote:

I see no problem with the scalar date scenarios listed below.

Thanks much, Patrick, for such a simple, sensible way of creating
single range start and end values, and Jan for your feedback that this
would work for you.

I've had many long discussions with Susan and Glen about this topic
and their frustrations with 'black box' or otherwise unsatisfactory
fuzzy date algorithms.  Using our current model / schema for
structured dates, with its separate Year/Month/Day fields for range
start/single date and end date - for which they both get considerable
credit - and the clear logic you outline below, date range search can
be transparent and easy to understand.

Note that entry of Year/Month/Day values versus uncertainty in the
structure date popup is an either/or, as I understand it.  If you want
the transparent, easy-to-understand behavior, you enter a year, and
maybe also a month, or month and day, for the start/single date, and
(optionally) for the end date.  If you enter uncertainty-related
values, when this is implemented down the road, perhaps a pluggable
fuzzy date handler would be invoked, a la Dick's suggestion?

Aron

On 8/24/11, Patrick Schmitz pschmitz@berkeley.edu wrote:

Maybe I am missing something, but is there a problem with the following:

Single date, expressed as "1910". We model this (in earliest and latest
scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910"

Date range expressed as earliest:"1910", latest:"1915". We model this as
esd: "Jan 1 1910" and lsd: "Dec 31 1915"

Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We model
this as esd: "Jan 1 1910" and lsd: "Aug 31 1910"

The esd and lsd scalar dates are used for search and ordering, not display.

Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf Of
Janice Eklund
Sent: Wednesday, August 24, 2011 10:17 AM
To: Susan Stone
Cc: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent structured dates?

I second Susan's concern about false precision.  At least 90%
of our dates are year only single dates or date ranges. If
you are considering fuzzy logic down the road, look at the
existing code carefully.  The fuzzy date machine, at least as
implemented at UCB for the HAVRC, does not handle ancient
dates properly.

Jan

On 8/24/11, Susan Stone sstone@socrates.berkeley.edu wrote:

Patrick,

There was quite a bit of discussion of this in our

Wednesday afternoon

meetings at Berkeley that I'm sure Aron will recall. I don't think
date fields go back far enough in time, so I'd recommend using
integers or floating point numbers and providing the math

in search.

It is also important to make sure that false precision isn't
introduced. If something is dated 1900, the system shouldn't enter
that as January 1, 1900 unless the uncertainty values can

be used to

differentiate and the display date can be simply 1900.

Susan

On 08/24/2011 09:16 AM, Patrick Schmitz wrote:

Really? I am a little surprised that in all the design discussions
for structured dates this did not come up.
Anyway, I will say that we do not have the time to build

some system

that allows for a lot of custom behavior, or elaborate heuristics
here (i.e., trying to divine what is intended from a set of text
fields). I would strongly suggest that we either

 1.
    require entry of some actual date fields upon which

the application

    can reason,
    or
 2.
    we define a very simple set of rules for building such

date values

    from some of the existing text fields. By very simple,

I mean that

    we only consider fields like year, month, day, and

that the rules

    produce two scalar dates (early and late), which

either define a

    range, or define a point (if they are equal). AIUI, a

structured

    date /cannot/ model the idea of "some time/any time after (or
    before) a date".

One further question is the semantic of which date (if

any) will be

used for ordering.
If and when advanced search supports date ranges (which

also implies

support for /before/ and /after/ logic in search), we can

handle the

interval logic associated with the structured dates, but

only if we

have a pair of scalar date values.
Down the road, I do think it would be interesting to

implement some

fuzzier logic on date search, to better support "circa"

logic and the

uncertainty values. However, I suspect that it will take longer to
arrive at consensus on these semantics, and would defer that
discussion for now.
Thanks - Patrick



    From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
    Sent: Wednesday, August 24, 2011 9:01 AM
    To: Patrick Schmitz
    Cc: 'Christopher Pott'; 'Aron Roberts';
talk@lists.collectionspace.org
    Subject: Re: [Talk] what's the best way to represent

structured

dates?

    Hi Patrick, I think it's fair to say there's no

consensus because

    there's been no discussion :-) It's possible that each

customer will

    want to use structured dates differently, which will

make it hard to

    add any intelligence or even predictability to things

like search.

    However, I also think that if a significant number of deploying
    institutions can come to some consensus on best ways to use and
    search structured dates, then many institutions will

choose to adopt

    the same best practices.

    Thanks,
    Chris

    On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote:

    Aron was out yesterday, and Rick is out for a few

days. They have

    done a lot of the structured dates work to date.
    The last I heard, Aron was hoping to get consensus

from the users

    about how we map all the text fields to actual date

fields that we

    can reason on (for search, ordering, etc). To date,

there seems to

    be no such consensus, which will preclude search,

ordering, etc.

    on structured dates. I would think that this would

make structured

    dates quite a bit less useful...
    Perhaps (I hope) my understanding is out of date, and

we just need

    to implement the logic. Perhaps Aron can jump in this

AM, when he

    gets in.
    Patrick



*On Behalf Of

        *Chris Hoffman
        Sent: Tuesday, August 23, 2011 11:05 AM
        To: Christopher Pott
        Cc: talk@lists.collectionspace.org
        mailto:talk@lists.collectionspace.org
        Subject: Re: [Talk] what's the best way to represent
        structured dates?

        Hi Chris,

        Thanks for asking these questions. We've been

wondering the

        same things. Have you changed any existing calendar date
        fields to structured dates or added new structured date
        fields? I think this is all quite new, and

together we might

        be able to come up with some best practices

(related to data

        migration and data entry).

        I suspect that search and advanced search of

structured date

        fields is going to be an issue for those of us deploying
        CollectionSpace. Our users will want CSpace to be

smart about

        structured dates, interpreting qualifiers, and so

on, but I

        bet that we won't see much as much as we would like in the
        first version of advanced search. I'm always happy to be
        proved wrong! :-)

        Chris

        On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote:

        Hi,

        I'm looking at the structured date schema and

the best way to

        represent dates migrated from our old system.

Any information

        about the following would be much appreciated..

        1. I see some different/overlapping possibilities for
        recording an object creation date. For example,

to represent

        "1718" (display date) one could either store:

        a) a single date, providing year only (1718)

        b) an earliest date (01.01.1718) with a latest date
(31.12.1718)

        c) a single date (01.01.1718) with a qualifier of + 364
days

        Is there any difference in the meaning of these

three (and

        would they all be treated equally by services such as
        advanced search)? Our migration data is

currently in format

        'b' but the other formats are sometimes

'friendlier' for new

        input.

        2. How are date "certainties" and "qualifiers" used by
        advanced search?

        I've specified "1718", as a creation date with a

certainty of

        "circa". Will a search for objects created

during 1717 result

        in a hit?

        Or is it necessary to use qualifiers (eg. 1718

+/- 2 years)

        to define exact limits for this type of searching?

        3. Is the Date schema locally extendable? (We

need a field to

        hold an English translation of the display text)

        4. Will Advanced search by creation date also search the
        display date or will it just use the single/earliest and
        latest date groups?

        Thanks,

        Chris

        _______________________________________________
        Talk mailing list
        Talk@lists.collectionspace.org
        mailto:Talk@lists.collectionspace.org

tionspace.org


Talk mailing list
Talk@lists.collectionspace.org

onspace.org


Talk mailing list
Talk@lists.collectionspace.org

nspace.org

On Wed, Aug 24, 2011 at 11:16 AM, Janice Eklund <janice.l.eklund@gmail.com> wrote: > I see no problem with the scalar date scenarios listed below. Thanks much, Patrick, for such a simple, sensible way of creating single range start and end values, and Jan for your feedback that this would work for you. I've had many long discussions with Susan and Glen about this topic and their frustrations with 'black box' or otherwise unsatisfactory fuzzy date algorithms. Using our current model / schema for structured dates, with its separate Year/Month/Day fields for range start/single date and end date - for which they both get considerable credit - and the clear logic you outline below, date range search can be transparent and easy to understand. Note that entry of Year/Month/Day values versus uncertainty in the structure date popup is an either/or, as I understand it. If you want the transparent, easy-to-understand behavior, you enter a year, and maybe also a month, or month and day, for the start/single date, and (optionally) for the end date. If you enter uncertainty-related values, when this is implemented down the road, perhaps a pluggable fuzzy date handler would be invoked, a la Dick's suggestion? Aron > On 8/24/11, Patrick Schmitz <pschmitz@berkeley.edu> wrote: >> Maybe I am missing something, but is there a problem with the following: >> >> Single date, expressed as "1910". We model this (in earliest and latest >> scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910" >> >> Date range expressed as earliest:"1910", latest:"1915". We model this as >> esd: "Jan 1 1910" and lsd: "Dec 31 1915" >> >> Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We model >> this as esd: "Jan 1 1910" and lsd: "Aug 31 1910" >> >> The esd and lsd scalar dates are used for search and ordering, not display. >> >> Patrick >> >>> -----Original Message----- >>> From: talk-bounces@lists.collectionspace.org >>> [mailto:talk-bounces@lists.collectionspace.org] On Behalf Of >>> Janice Eklund >>> Sent: Wednesday, August 24, 2011 10:17 AM >>> To: Susan Stone >>> Cc: talk@lists.collectionspace.org >>> Subject: Re: [Talk] what's the best way to represent structured dates? >>> >>> I second Susan's concern about false precision.  At least 90% >>> of our dates are year only single dates or date ranges. If >>> you are considering fuzzy logic down the road, look at the >>> existing code carefully.  The fuzzy date machine, at least as >>> implemented at UCB for the HAVRC, does not handle ancient >>> dates properly. >>> >>> Jan >>> >>> On 8/24/11, Susan Stone <sstone@socrates.berkeley.edu> wrote: >>> > Patrick, >>> > >>> > There was quite a bit of discussion of this in our >>> Wednesday afternoon >>> > meetings at Berkeley that I'm sure Aron will recall. I don't think >>> > date fields go back far enough in time, so I'd recommend using >>> > integers or floating point numbers and providing the math >>> in search. >>> > It is also important to make sure that false precision isn't >>> > introduced. If something is dated 1900, the system shouldn't enter >>> > that as January 1, 1900 unless the uncertainty values can >>> be used to >>> > differentiate and the display date can be simply 1900. >>> > >>> > Susan >>> > >>> > >>> > On 08/24/2011 09:16 AM, Patrick Schmitz wrote: >>> >> Really? I am a little surprised that in all the design discussions >>> >> for structured dates this did not come up. >>> >> Anyway, I will say that we do not have the time to build >>> some system >>> >> that allows for a lot of custom behavior, or elaborate heuristics >>> >> here (i.e., trying to divine what is intended from a set of text >>> >> fields). I would strongly suggest that we either >>> >> >>> >>  1. >>> >>     require entry of some actual date fields upon which >>> the application >>> >>     can reason, >>> >>     or >>> >>  2. >>> >>     we define a very simple set of rules for building such >>> date values >>> >>     from some of the existing text fields. By very simple, >>> I mean that >>> >>     we only consider fields like year, month, day, and >>> that the rules >>> >>     produce two scalar dates (early and late), which >>> either define a >>> >>     range, or define a point (if they are equal). AIUI, a >>> structured >>> >>     date /cannot/ model the idea of "some time/any time after (or >>> >>     before) a date". >>> >> >>> >> One further question is the semantic of which date (if >>> any) will be >>> >> used for ordering. >>> >> If and when advanced search supports date ranges (which >>> also implies >>> >> support for /before/ and /after/ logic in search), we can >>> handle the >>> >> interval logic associated with the structured dates, but >>> only if we >>> >> have a pair of scalar date values. >>> >> Down the road, I do think it would be interesting to >>> implement some >>> >> fuzzier logic on date search, to better support "circa" >>> logic and the >>> >> uncertainty values. However, I suspect that it will take longer to >>> >> arrive at consensus on these semantics, and would defer that >>> >> discussion for now. >>> >> Thanks - Patrick >>> >> >>> >> >>> >> >>> -------------------------------------------------------------- >>> ---------- >>> >>     *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu] >>> >>     *Sent:* Wednesday, August 24, 2011 9:01 AM >>> >>     *To:* Patrick Schmitz >>> >>     *Cc:* 'Christopher Pott'; 'Aron Roberts'; >>> >> talk@lists.collectionspace.org >>> >>     *Subject:* Re: [Talk] what's the best way to represent >>> structured >>> >> dates? >>> >> >>> >>     Hi Patrick, I think it's fair to say there's no >>> consensus because >>> >>     there's been no discussion :-) It's possible that each >>> customer will >>> >>     want to use structured dates differently, which will >>> make it hard to >>> >>     add any intelligence or even predictability to things >>> like search. >>> >>     However, I also think that if a significant number of deploying >>> >>     institutions can come to some consensus on best ways to use and >>> >>     search structured dates, then many institutions will >>> choose to adopt >>> >>     the same best practices. >>> >> >>> >>     Thanks, >>> >>     Chris >>> >> >>> >>     On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote: >>> >> >>> >>>     Aron was out yesterday, and Rick is out for a few >>> days. They have >>> >>>     done a lot of the structured dates work to date. >>> >>>     The last I heard, Aron was hoping to get consensus >>> from the users >>> >>>     about how we map all the text fields to actual date >>> fields that we >>> >>>     can reason on (for search, ordering, etc). To date, >>> there seems to >>> >>>     be no such consensus, which will preclude search, >>> ordering, etc. >>> >>>     on structured dates. I would think that this would >>> make structured >>> >>>     dates quite a bit less useful... >>> >>>     Perhaps (I hope) my understanding is out of date, and >>> we just need >>> >>>     to implement the logic. Perhaps Aron can jump in this >>> AM, when he >>> >>>     gets in. >>> >>>     Patrick >>> >>> >>> >>> >>> >>> >>> -------------------------------------------------------------- >>> ---------- >>> >>>         *From:* talk-bounces@lists.collectionspace.org >>> >>>         <mailto:talk-bounces@lists.collectionspace.org> >>> >>>         [mailto:talk-bounces@lists.collectionspace.org] >>> *On Behalf Of >>> >>>         *Chris Hoffman >>> >>>         *Sent:* Tuesday, August 23, 2011 11:05 AM >>> >>>         *To:* Christopher Pott >>> >>>         *Cc:* talk@lists.collectionspace.org >>> >>>         <mailto:talk@lists.collectionspace.org> >>> >>>         *Subject:* Re: [Talk] what's the best way to represent >>> >>>         structured dates? >>> >>> >>> >>>         Hi Chris, >>> >>> >>> >>>         Thanks for asking these questions. We've been >>> wondering the >>> >>>         same things. Have you changed any existing calendar date >>> >>>         fields to structured dates or added new structured date >>> >>>         fields? I think this is all quite new, and >>> together we might >>> >>>         be able to come up with some best practices >>> (related to data >>> >>>         migration and data entry). >>> >>> >>> >>>         I suspect that search and advanced search of >>> structured date >>> >>>         fields is going to be an issue for those of us deploying >>> >>>         CollectionSpace. Our users will want CSpace to be >>> smart about >>> >>>         structured dates, interpreting qualifiers, and so >>> on, but I >>> >>>         bet that we won't see much as much as we would like in the >>> >>>         first version of advanced search. I'm always happy to be >>> >>>         proved wrong! :-) >>> >>> >>> >>>         Chris >>> >>> >>> >>>         On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote: >>> >>> >>> >>>>         Hi, >>> >>>> >>> >>>>         I'm looking at the structured date schema and >>> the best way to >>> >>>>         represent dates migrated from our old system. >>> Any information >>> >>>>         about the following would be much appreciated.. >>> >>>> >>> >>>>         1. I see some different/overlapping possibilities for >>> >>>>         recording an object creation date. For example, >>> to represent >>> >>>>         "1718" (display date) one could either store: >>> >>>> >>> >>>>         a) a single date, providing year only (1718) >>> >>>> >>> >>>>         b) an earliest date (01.01.1718) with a latest date >>> >>>> (31.12.1718) >>> >>>> >>> >>>>         c) a single date (01.01.1718) with a qualifier of + 364 >>> >>>> days >>> >>>> >>> >>>>         Is there any difference in the meaning of these >>> three (and >>> >>>>         would they all be treated equally by services such as >>> >>>>         advanced search)? Our migration data is >>> currently in format >>> >>>>         'b' but the other formats are sometimes >>> 'friendlier' for new >>> >>>>         input. >>> >>>> >>> >>>>         2. How are date "certainties" and "qualifiers" used by >>> >>>>         advanced search? >>> >>>> >>> >>>>         I've specified "1718", as a creation date with a >>> certainty of >>> >>>>         "circa". Will a search for objects created >>> during 1717 result >>> >>>>         in a hit? >>> >>>> >>> >>>>         Or is it necessary to use qualifiers (eg. 1718 >>> +/- 2 years) >>> >>>>         to define exact limits for this type of searching? >>> >>>> >>> >>>>         3. Is the Date schema locally extendable? (We >>> need a field to >>> >>>>         hold an English translation of the display text) >>> >>>> >>> >>>>         4. Will Advanced search by creation date also search the >>> >>>>         display date or will it just use the single/earliest and >>> >>>>         latest date groups? >>> >>>> >>> >>>>         Thanks, >>> >>>> >>> >>>>         Chris >>> >>>> >>> >>>>         _______________________________________________ >>> >>>>         Talk mailing list >>> >>>>         Talk@lists.collectionspace.org >>> >>>>         <mailto:Talk@lists.collectionspace.org> >>> >>>> >>> >>>> >>> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collec >>> >>>> tionspace.org >>> >>> >>> >> >>> >> >>> >> >>> >> _______________________________________________ >>> >> Talk mailing list >>> >> Talk@lists.collectionspace.org >>> >> >>> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collecti >>> >> onspace.org >>> > >>> > >>> > _______________________________________________ >>> > Talk mailing list >>> > Talk@lists.collectionspace.org >>> > >>> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectio >>> > nspace.org >>> > >>> >>> _______________________________________________ >>> Talk mailing list >>> Talk@lists.collectionspace.org >>> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c >>> ollectionspace.org >>> >> >> > > _______________________________________________ > Talk mailing list > Talk@lists.collectionspace.org > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspace.org >
PS
Patrick Schmitz
Wed, Aug 24, 2011 7:00 PM

I think there is lots of potential for fuzzy date interpretation, and we can
pursue this as a second step.

Next, we need to find out if the 300M yr BCE range will work for Michael,
for Paleo, Essig, et al. I am worried about Paleo in particular.

Any other folks on this list with collections that go back a long way? How
much of a limitation would the 300M yr BCE range be?

Thanks - Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf Of
Aron Roberts
Sent: Wednesday, August 24, 2011 11:39 AM
To: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent structured dates?

On Wed, Aug 24, 2011 at 11:16 AM, Janice Eklund
janice.l.eklund@gmail.com wrote:

I see no problem with the scalar date scenarios listed below.

Thanks much, Patrick, for such a simple, sensible way of
creating single range start and end values, and Jan for your
feedback that this would work for you.

I've had many long discussions with Susan and Glen about
this topic and their frustrations with 'black box' or
otherwise unsatisfactory fuzzy date algorithms.  Using our
current model / schema for structured dates, with its
separate Year/Month/Day fields for range start/single date
and end date - for which they both get considerable credit -
and the clear logic you outline below, date range search can
be transparent and easy to understand.

Note that entry of Year/Month/Day values versus uncertainty
in the structure date popup is an either/or, as I understand
it.  If you want the transparent, easy-to-understand
behavior, you enter a year, and maybe also a month, or month
and day, for the start/single date, and
(optionally) for the end date.  If you enter uncertainty-related
values, when this is implemented down the road, perhaps a
pluggable fuzzy date handler would be invoked, a la Dick's suggestion?

Aron

On 8/24/11, Patrick Schmitz pschmitz@berkeley.edu wrote:

Maybe I am missing something, but is there a problem with

the following:

Single date, expressed as "1910". We model this (in earliest and
latest scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910"

Date range expressed as earliest:"1910", latest:"1915". We

model this

as
esd: "Jan 1 1910" and lsd: "Dec 31 1915"

Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We
model this as esd: "Jan 1 1910" and lsd: "Aug 31 1910"

The esd and lsd scalar dates are used for search and

ordering, not display.

Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf

Of Janice

Eklund
Sent: Wednesday, August 24, 2011 10:17 AM
To: Susan Stone
Cc: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured dates?

I second Susan's concern about false precision.  At least

90% of our

dates are year only single dates or date ranges. If you are
considering fuzzy logic down the road, look at the existing code
carefully.  The fuzzy date machine, at least as

implemented at UCB

for the HAVRC, does not handle ancient dates properly.

Jan

On 8/24/11, Susan Stone sstone@socrates.berkeley.edu wrote:

Patrick,

There was quite a bit of discussion of this in our

Wednesday afternoon

meetings at Berkeley that I'm sure Aron will recall. I

don't think

date fields go back far enough in time, so I'd recommend using
integers or floating point numbers and providing the math

in search.

It is also important to make sure that false precision isn't
introduced. If something is dated 1900, the system

shouldn't enter

that as January 1, 1900 unless the uncertainty values can

be used to

differentiate and the display date can be simply 1900.

Susan

On 08/24/2011 09:16 AM, Patrick Schmitz wrote:

Really? I am a little surprised that in all the design
discussions for structured dates this did not come up.
Anyway, I will say that we do not have the time to build

some system

that allows for a lot of custom behavior, or elaborate

heuristics

here (i.e., trying to divine what is intended from a

set of text

fields). I would strongly suggest that we either

 1.
    require entry of some actual date fields upon which

the application

    can reason,
    or
 2.
    we define a very simple set of rules for building such

date values

    from some of the existing text fields. By very simple,

I mean that

    we only consider fields like year, month, day, and

that the rules

    produce two scalar dates (early and late), which

either define a

    range, or define a point (if they are equal). AIUI, a

structured

    date /cannot/ model the idea of "some time/any

time after (or

    before) a date".

One further question is the semantic of which date (if

any) will be

used for ordering.
If and when advanced search supports date ranges (which

also implies

support for /before/ and /after/ logic in search), we can

handle the

interval logic associated with the structured dates, but

only if we

have a pair of scalar date values.
Down the road, I do think it would be interesting to

implement some

fuzzier logic on date search, to better support "circa"

logic and the

uncertainty values. However, I suspect that it will

take longer

to arrive at consensus on these semantics, and would

defer that

discussion for now.
Thanks - Patrick



    From: Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
    Sent: Wednesday, August 24, 2011 9:01 AM
    To: Patrick Schmitz
    Cc: 'Christopher Pott'; 'Aron Roberts';
talk@lists.collectionspace.org
    Subject: Re: [Talk] what's the best way to represent

structured

dates?

    Hi Patrick, I think it's fair to say there's no

consensus because

    there's been no discussion :-) It's possible that each

customer will

    want to use structured dates differently, which will

make it hard to

    add any intelligence or even predictability to things

like search.

    However, I also think that if a significant number of
deploying
    institutions can come to some consensus on best

ways to use

and
    search structured dates, then many institutions will

choose to adopt

    the same best practices.

    Thanks,
    Chris

    On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote:

    Aron was out yesterday, and Rick is out for a few

days. They have

    done a lot of the structured dates work to date.
    The last I heard, Aron was hoping to get consensus

from the users

    about how we map all the text fields to actual date

fields that we

    can reason on (for search, ordering, etc). To date,

there seems to

    be no such consensus, which will preclude search,

ordering, etc.

    on structured dates. I would think that this would

make structured

    dates quite a bit less useful...
    Perhaps (I hope) my understanding is out of date, and

we just need

    to implement the logic. Perhaps Aron can jump in this

AM, when he

    gets in.
    Patrick



*On Behalf Of

        *Chris Hoffman
        Sent: Tuesday, August 23, 2011 11:05 AM
        To: Christopher Pott
        Cc: talk@lists.collectionspace.org
        mailto:talk@lists.collectionspace.org
        Subject: Re: [Talk] what's the best way to represent
        structured dates?

        Hi Chris,

        Thanks for asking these questions. We've been

wondering the

        same things. Have you changed any existing

calendar date

        fields to structured dates or added new

structured date

        fields? I think this is all quite new, and

together we might

        be able to come up with some best practices

(related to data

        migration and data entry).

        I suspect that search and advanced search of

structured date

        fields is going to be an issue for those of

us deploying

        CollectionSpace. Our users will want CSpace to be

smart about

        structured dates, interpreting qualifiers, and so

on, but I

        bet that we won't see much as much as we

would like in

the
        first version of advanced search. I'm always

happy to be

        proved wrong! :-)

        Chris

        On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote:

        Hi,

        I'm looking at the structured date schema and

the best way to

        represent dates migrated from our old system.

Any information

        about the following would be much appreciated..

        1. I see some different/overlapping possibilities for
        recording an object creation date. For example,

to represent

        "1718" (display date) one could either store:

        a) a single date, providing year only (1718)

        b) an earliest date (01.01.1718) with a latest date
(31.12.1718)

        c) a single date (01.01.1718) with a

qualifier of + 364

days

        Is there any difference in the meaning of these

three (and

        would they all be treated equally by services such as
        advanced search)? Our migration data is

currently in format

        'b' but the other formats are sometimes

'friendlier' for new

        input.

        2. How are date "certainties" and

"qualifiers" used by

        advanced search?

        I've specified "1718", as a creation date with a

certainty of

        "circa". Will a search for objects created

during 1717 result

        in a hit?

        Or is it necessary to use qualifiers (eg. 1718

+/- 2 years)

        to define exact limits for this type of searching?

        3. Is the Date schema locally extendable? (We

need a field to

        hold an English translation of the display text)

        4. Will Advanced search by creation date also search
the
        display date or will it just use the single/earliest
and
        latest date groups?

        Thanks,

        Chris

        _______________________________________________
        Talk mailing list
        Talk@lists.collectionspace.org
        mailto:Talk@lists.collectionspace.org

tionspace.org


Talk mailing list
Talk@lists.collectionspace.org

i

onspace.org


Talk mailing list
Talk@lists.collectionspace.org

nspace.org

I think there is lots of potential for fuzzy date interpretation, and we can pursue this as a second step. Next, we need to find out if the 300M yr BCE range will work for Michael, for Paleo, Essig, et al. I am worried about Paleo in particular. Any other folks on this list with collections that go back a long way? How much of a limitation would the 300M yr BCE range be? Thanks - Patrick > -----Original Message----- > From: talk-bounces@lists.collectionspace.org > [mailto:talk-bounces@lists.collectionspace.org] On Behalf Of > Aron Roberts > Sent: Wednesday, August 24, 2011 11:39 AM > To: talk@lists.collectionspace.org > Subject: Re: [Talk] what's the best way to represent structured dates? > > On Wed, Aug 24, 2011 at 11:16 AM, Janice Eklund > <janice.l.eklund@gmail.com> wrote: > > I see no problem with the scalar date scenarios listed below. > > Thanks much, Patrick, for such a simple, sensible way of > creating single range start and end values, and Jan for your > feedback that this would work for you. > > I've had many long discussions with Susan and Glen about > this topic and their frustrations with 'black box' or > otherwise unsatisfactory fuzzy date algorithms. Using our > current model / schema for structured dates, with its > separate Year/Month/Day fields for range start/single date > and end date - for which they both get considerable credit - > and the clear logic you outline below, date range search can > be transparent and easy to understand. > > Note that entry of Year/Month/Day values versus uncertainty > in the structure date popup is an either/or, as I understand > it. If you want the transparent, easy-to-understand > behavior, you enter a year, and maybe also a month, or month > and day, for the start/single date, and > (optionally) for the end date. If you enter uncertainty-related > values, when this is implemented down the road, perhaps a > pluggable fuzzy date handler would be invoked, a la Dick's suggestion? > > Aron > > > On 8/24/11, Patrick Schmitz <pschmitz@berkeley.edu> wrote: > >> Maybe I am missing something, but is there a problem with > the following: > >> > >> Single date, expressed as "1910". We model this (in earliest and > >> latest scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910" > >> > >> Date range expressed as earliest:"1910", latest:"1915". We > model this > >> as > >> esd: "Jan 1 1910" and lsd: "Dec 31 1915" > >> > >> Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We > >> model this as esd: "Jan 1 1910" and lsd: "Aug 31 1910" > >> > >> The esd and lsd scalar dates are used for search and > ordering, not display. > >> > >> Patrick > >> > >>> -----Original Message----- > >>> From: talk-bounces@lists.collectionspace.org > >>> [mailto:talk-bounces@lists.collectionspace.org] On Behalf > Of Janice > >>> Eklund > >>> Sent: Wednesday, August 24, 2011 10:17 AM > >>> To: Susan Stone > >>> Cc: talk@lists.collectionspace.org > >>> Subject: Re: [Talk] what's the best way to represent > structured dates? > >>> > >>> I second Susan's concern about false precision.  At least > 90% of our > >>> dates are year only single dates or date ranges. If you are > >>> considering fuzzy logic down the road, look at the existing code > >>> carefully.  The fuzzy date machine, at least as > implemented at UCB > >>> for the HAVRC, does not handle ancient dates properly. > >>> > >>> Jan > >>> > >>> On 8/24/11, Susan Stone <sstone@socrates.berkeley.edu> wrote: > >>> > Patrick, > >>> > > >>> > There was quite a bit of discussion of this in our > >>> Wednesday afternoon > >>> > meetings at Berkeley that I'm sure Aron will recall. I > don't think > >>> > date fields go back far enough in time, so I'd recommend using > >>> > integers or floating point numbers and providing the math > >>> in search. > >>> > It is also important to make sure that false precision isn't > >>> > introduced. If something is dated 1900, the system > shouldn't enter > >>> > that as January 1, 1900 unless the uncertainty values can > >>> be used to > >>> > differentiate and the display date can be simply 1900. > >>> > > >>> > Susan > >>> > > >>> > > >>> > On 08/24/2011 09:16 AM, Patrick Schmitz wrote: > >>> >> Really? I am a little surprised that in all the design > >>> >> discussions for structured dates this did not come up. > >>> >> Anyway, I will say that we do not have the time to build > >>> some system > >>> >> that allows for a lot of custom behavior, or elaborate > heuristics > >>> >> here (i.e., trying to divine what is intended from a > set of text > >>> >> fields). I would strongly suggest that we either > >>> >> > >>> >>  1. > >>> >>     require entry of some actual date fields upon which > >>> the application > >>> >>     can reason, > >>> >>     or > >>> >>  2. > >>> >>     we define a very simple set of rules for building such > >>> date values > >>> >>     from some of the existing text fields. By very simple, > >>> I mean that > >>> >>     we only consider fields like year, month, day, and > >>> that the rules > >>> >>     produce two scalar dates (early and late), which > >>> either define a > >>> >>     range, or define a point (if they are equal). AIUI, a > >>> structured > >>> >>     date /cannot/ model the idea of "some time/any > time after (or > >>> >>     before) a date". > >>> >> > >>> >> One further question is the semantic of which date (if > >>> any) will be > >>> >> used for ordering. > >>> >> If and when advanced search supports date ranges (which > >>> also implies > >>> >> support for /before/ and /after/ logic in search), we can > >>> handle the > >>> >> interval logic associated with the structured dates, but > >>> only if we > >>> >> have a pair of scalar date values. > >>> >> Down the road, I do think it would be interesting to > >>> implement some > >>> >> fuzzier logic on date search, to better support "circa" > >>> logic and the > >>> >> uncertainty values. However, I suspect that it will > take longer > >>> >> to arrive at consensus on these semantics, and would > defer that > >>> >> discussion for now. > >>> >> Thanks - Patrick > >>> >> > >>> >> > >>> >> > >>> -------------------------------------------------------------- > >>> ---------- > >>> >>     *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu] > >>> >>     *Sent:* Wednesday, August 24, 2011 9:01 AM > >>> >>     *To:* Patrick Schmitz > >>> >>     *Cc:* 'Christopher Pott'; 'Aron Roberts'; > >>> >> talk@lists.collectionspace.org > >>> >>     *Subject:* Re: [Talk] what's the best way to represent > >>> structured > >>> >> dates? > >>> >> > >>> >>     Hi Patrick, I think it's fair to say there's no > >>> consensus because > >>> >>     there's been no discussion :-) It's possible that each > >>> customer will > >>> >>     want to use structured dates differently, which will > >>> make it hard to > >>> >>     add any intelligence or even predictability to things > >>> like search. > >>> >>     However, I also think that if a significant number of > >>> >> deploying > >>> >>     institutions can come to some consensus on best > ways to use > >>> >> and > >>> >>     search structured dates, then many institutions will > >>> choose to adopt > >>> >>     the same best practices. > >>> >> > >>> >>     Thanks, > >>> >>     Chris > >>> >> > >>> >>     On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote: > >>> >> > >>> >>>     Aron was out yesterday, and Rick is out for a few > >>> days. They have > >>> >>>     done a lot of the structured dates work to date. > >>> >>>     The last I heard, Aron was hoping to get consensus > >>> from the users > >>> >>>     about how we map all the text fields to actual date > >>> fields that we > >>> >>>     can reason on (for search, ordering, etc). To date, > >>> there seems to > >>> >>>     be no such consensus, which will preclude search, > >>> ordering, etc. > >>> >>>     on structured dates. I would think that this would > >>> make structured > >>> >>>     dates quite a bit less useful... > >>> >>>     Perhaps (I hope) my understanding is out of date, and > >>> we just need > >>> >>>     to implement the logic. Perhaps Aron can jump in this > >>> AM, when he > >>> >>>     gets in. > >>> >>>     Patrick > >>> >>> > >>> >>> > >>> >>> > >>> -------------------------------------------------------------- > >>> ---------- > >>> >>>         *From:* talk-bounces@lists.collectionspace.org > >>> >>>         <mailto:talk-bounces@lists.collectionspace.org> > >>> >>>         [mailto:talk-bounces@lists.collectionspace.org] > >>> *On Behalf Of > >>> >>>         *Chris Hoffman > >>> >>>         *Sent:* Tuesday, August 23, 2011 11:05 AM > >>> >>>         *To:* Christopher Pott > >>> >>>         *Cc:* talk@lists.collectionspace.org > >>> >>>         <mailto:talk@lists.collectionspace.org> > >>> >>>         *Subject:* Re: [Talk] what's the best way to represent > >>> >>>         structured dates? > >>> >>> > >>> >>>         Hi Chris, > >>> >>> > >>> >>>         Thanks for asking these questions. We've been > >>> wondering the > >>> >>>         same things. Have you changed any existing > calendar date > >>> >>>         fields to structured dates or added new > structured date > >>> >>>         fields? I think this is all quite new, and > >>> together we might > >>> >>>         be able to come up with some best practices > >>> (related to data > >>> >>>         migration and data entry). > >>> >>> > >>> >>>         I suspect that search and advanced search of > >>> structured date > >>> >>>         fields is going to be an issue for those of > us deploying > >>> >>>         CollectionSpace. Our users will want CSpace to be > >>> smart about > >>> >>>         structured dates, interpreting qualifiers, and so > >>> on, but I > >>> >>>         bet that we won't see much as much as we > would like in > >>> >>> the > >>> >>>         first version of advanced search. I'm always > happy to be > >>> >>>         proved wrong! :-) > >>> >>> > >>> >>>         Chris > >>> >>> > >>> >>>         On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote: > >>> >>> > >>> >>>>         Hi, > >>> >>>> > >>> >>>>         I'm looking at the structured date schema and > >>> the best way to > >>> >>>>         represent dates migrated from our old system. > >>> Any information > >>> >>>>         about the following would be much appreciated.. > >>> >>>> > >>> >>>>         1. I see some different/overlapping possibilities for > >>> >>>>         recording an object creation date. For example, > >>> to represent > >>> >>>>         "1718" (display date) one could either store: > >>> >>>> > >>> >>>>         a) a single date, providing year only (1718) > >>> >>>> > >>> >>>>         b) an earliest date (01.01.1718) with a latest date > >>> >>>> (31.12.1718) > >>> >>>> > >>> >>>>         c) a single date (01.01.1718) with a > qualifier of + 364 > >>> >>>> days > >>> >>>> > >>> >>>>         Is there any difference in the meaning of these > >>> three (and > >>> >>>>         would they all be treated equally by services such as > >>> >>>>         advanced search)? Our migration data is > >>> currently in format > >>> >>>>         'b' but the other formats are sometimes > >>> 'friendlier' for new > >>> >>>>         input. > >>> >>>> > >>> >>>>         2. How are date "certainties" and > "qualifiers" used by > >>> >>>>         advanced search? > >>> >>>> > >>> >>>>         I've specified "1718", as a creation date with a > >>> certainty of > >>> >>>>         "circa". Will a search for objects created > >>> during 1717 result > >>> >>>>         in a hit? > >>> >>>> > >>> >>>>         Or is it necessary to use qualifiers (eg. 1718 > >>> +/- 2 years) > >>> >>>>         to define exact limits for this type of searching? > >>> >>>> > >>> >>>>         3. Is the Date schema locally extendable? (We > >>> need a field to > >>> >>>>         hold an English translation of the display text) > >>> >>>> > >>> >>>>         4. Will Advanced search by creation date also search > >>> >>>> the > >>> >>>>         display date or will it just use the single/earliest > >>> >>>> and > >>> >>>>         latest date groups? > >>> >>>> > >>> >>>>         Thanks, > >>> >>>> > >>> >>>>         Chris > >>> >>>> > >>> >>>>         _______________________________________________ > >>> >>>>         Talk mailing list > >>> >>>>         Talk@lists.collectionspace.org > >>> >>>>         <mailto:Talk@lists.collectionspace.org> > >>> >>>> > >>> >>>> > >>> > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collec > >>> >>>> tionspace.org > >>> >>> > >>> >> > >>> >> > >>> >> > >>> >> _______________________________________________ > >>> >> Talk mailing list > >>> >> Talk@lists.collectionspace.org > >>> >> > >>> > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collect > >>> i > >>> >> onspace.org > >>> > > >>> > > >>> > _______________________________________________ > >>> > Talk mailing list > >>> > Talk@lists.collectionspace.org > >>> > > >>> > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collect > >>> io > >>> > nspace.org > >>> > > >>> > >>> _______________________________________________ > >>> Talk mailing list > >>> Talk@lists.collectionspace.org > >>> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c > >>> ollectionspace.org > >>> > >> > >> > > > > _______________________________________________ > > Talk mailing list > > Talk@lists.collectionspace.org > > > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectio > > nspace.org > > > > _______________________________________________ > Talk mailing list > Talk@lists.collectionspace.org > http://lists.collectionspace.org/mailman/listinfo/talk_lists.c > ollectionspace.org >
SS
Susan Stone
Wed, Aug 24, 2011 7:47 PM

This is fine for search and sort.

Susan

On 08/24/2011 11:05 AM, Patrick Schmitz wrote:

Maybe I am missing something, but is there a problem with the following:

Single date, expressed as "1910". We model this (in earliest and latest
scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910"

Date range expressed as earliest:"1910", latest:"1915". We model this as
esd: "Jan 1 1910" and lsd: "Dec 31 1915"

Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We model
this as esd: "Jan 1 1910" and lsd: "Aug 31 1910"

The esd and lsd scalar dates are used for search and ordering, not display.

Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf Of
Janice Eklund
Sent: Wednesday, August 24, 2011 10:17 AM
To: Susan Stone
Cc: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent structured dates?

I second Susan's concern about false precision.  At least 90%
of our dates are year only single dates or date ranges. If
you are considering fuzzy logic down the road, look at the
existing code carefully.  The fuzzy date machine, at least as
implemented at UCB for the HAVRC, does not handle ancient
dates properly.

Jan

On 8/24/11, Susan Stonesstone@socrates.berkeley.edu  wrote:

Patrick,

There was quite a bit of discussion of this in our

Wednesday afternoon

meetings at Berkeley that I'm sure Aron will recall. I don't think
date fields go back far enough in time, so I'd recommend using
integers or floating point numbers and providing the math

in search.

It is also important to make sure that false precision isn't
introduced. If something is dated 1900, the system shouldn't enter
that as January 1, 1900 unless the uncertainty values can

be used to

differentiate and the display date can be simply 1900.

Susan

On 08/24/2011 09:16 AM, Patrick Schmitz wrote:

Really? I am a little surprised that in all the design discussions
for structured dates this did not come up.
Anyway, I will say that we do not have the time to build

some system

that allows for a lot of custom behavior, or elaborate heuristics
here (i.e., trying to divine what is intended from a set of text
fields). I would strongly suggest that we either

  1. require entry of some actual date fields upon which

the application

  can reason,
  or
  1. we define a very simple set of rules for building such

date values

  from some of the existing text fields. By very simple,

I mean that

  we only consider fields like year, month, day, and

that the rules

  produce two scalar dates (early and late), which

either define a

  range, or define a point (if they are equal). AIUI, a

structured

  date /cannot/ model the idea of "some time/any time after (or
  before) a date".

One further question is the semantic of which date (if

any) will be

used for ordering.
If and when advanced search supports date ranges (which

also implies

support for /before/ and /after/ logic in search), we can

handle the

interval logic associated with the structured dates, but

only if we

have a pair of scalar date values.
Down the road, I do think it would be interesting to

implement some

fuzzier logic on date search, to better support "circa"

logic and the

uncertainty values. However, I suspect that it will take longer to
arrive at consensus on these semantics, and would defer that
discussion for now.
Thanks - Patrick



  *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
  *Sent:* Wednesday, August 24, 2011 9:01 AM
  *To:* Patrick Schmitz
  *Cc:* 'Christopher Pott'; 'Aron Roberts';

talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured

dates?

  Hi Patrick, I think it's fair to say there's no

consensus because

  there's been no discussion :-) It's possible that each

customer will

  want to use structured dates differently, which will

make it hard to

  add any intelligence or even predictability to things

like search.

  However, I also think that if a significant number of deploying
  institutions can come to some consensus on best ways to use and
  search structured dates, then many institutions will

choose to adopt

  the same best practices.

  Thanks,
  Chris

  On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote:
  Aron was out yesterday, and Rick is out for a few

days. They have

  done a lot of the structured dates work to date.
  The last I heard, Aron was hoping to get consensus

from the users

  about how we map all the text fields to actual date

fields that we

  can reason on (for search, ordering, etc). To date,

there seems to

  be no such consensus, which will preclude search,

ordering, etc.

  on structured dates. I would think that this would

make structured

  dates quite a bit less useful...
  Perhaps (I hope) my understanding is out of date, and

we just need

  to implement the logic. Perhaps Aron can jump in this

AM, when he

  gets in.
  Patrick


      *From:* talk-bounces@lists.collectionspace.org
      <mailto:talk-bounces@lists.collectionspace.org>
      [mailto:talk-bounces@lists.collectionspace.org]

*On Behalf Of

      *Chris Hoffman
      *Sent:* Tuesday, August 23, 2011 11:05 AM
      *To:* Christopher Pott
      *Cc:* talk@lists.collectionspace.org
      <mailto:talk@lists.collectionspace.org>
      *Subject:* Re: [Talk] what's the best way to represent
      structured dates?

      Hi Chris,

      Thanks for asking these questions. We've been

wondering the

      same things. Have you changed any existing calendar date
      fields to structured dates or added new structured date
      fields? I think this is all quite new, and

together we might

      be able to come up with some best practices

(related to data

      migration and data entry).

      I suspect that search and advanced search of

structured date

      fields is going to be an issue for those of us deploying
      CollectionSpace. Our users will want CSpace to be

smart about

      structured dates, interpreting qualifiers, and so

on, but I

      bet that we won't see much as much as we would like in the
      first version of advanced search. I'm always happy to be
      proved wrong! :-)

      Chris

      On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote:
      Hi,

      I'm looking at the structured date schema and

the best way to

      represent dates migrated from our old system.

Any information

      about the following would be much appreciated..

      1. I see some different/overlapping possibilities for
      recording an object creation date. For example,

to represent

      "1718" (display date) one could either store:

      a) a single date, providing year only (1718)

      b) an earliest date (01.01.1718) with a latest date

(31.12.1718)

      c) a single date (01.01.1718) with a qualifier of + 364

days

      Is there any difference in the meaning of these

three (and

      would they all be treated equally by services such as
      advanced search)? Our migration data is

currently in format

      'b' but the other formats are sometimes

'friendlier' for new

      input.

      2. How are date "certainties" and "qualifiers" used by
      advanced search?

      I've specified "1718", as a creation date with a

certainty of

      "circa". Will a search for objects created

during 1717 result

      in a hit?

      Or is it necessary to use qualifiers (eg. 1718

+/- 2 years)

      to define exact limits for this type of searching?

      3. Is the Date schema locally extendable? (We

need a field to

      hold an English translation of the display text)

      4. Will Advanced search by creation date also search the
      display date or will it just use the single/earliest and
      latest date groups?

      Thanks,

      Chris

      _______________________________________________
      Talk mailing list
      Talk@lists.collectionspace.org
      <mailto:Talk@lists.collectionspace.org>

tionspace.org


Talk mailing list
Talk@lists.collectionspace.org

onspace.org


Talk mailing list
Talk@lists.collectionspace.org

nspace.org

This is fine for search and sort. Susan On 08/24/2011 11:05 AM, Patrick Schmitz wrote: > Maybe I am missing something, but is there a problem with the following: > > Single date, expressed as "1910". We model this (in earliest and latest > scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910" > > Date range expressed as earliest:"1910", latest:"1915". We model this as > esd: "Jan 1 1910" and lsd: "Dec 31 1915" > > Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We model > this as esd: "Jan 1 1910" and lsd: "Aug 31 1910" > > The esd and lsd scalar dates are used for search and ordering, not display. > > Patrick > >> -----Original Message----- >> From: talk-bounces@lists.collectionspace.org >> [mailto:talk-bounces@lists.collectionspace.org] On Behalf Of >> Janice Eklund >> Sent: Wednesday, August 24, 2011 10:17 AM >> To: Susan Stone >> Cc: talk@lists.collectionspace.org >> Subject: Re: [Talk] what's the best way to represent structured dates? >> >> I second Susan's concern about false precision. At least 90% >> of our dates are year only single dates or date ranges. If >> you are considering fuzzy logic down the road, look at the >> existing code carefully. The fuzzy date machine, at least as >> implemented at UCB for the HAVRC, does not handle ancient >> dates properly. >> >> Jan >> >> On 8/24/11, Susan Stone<sstone@socrates.berkeley.edu> wrote: >>> Patrick, >>> >>> There was quite a bit of discussion of this in our >> Wednesday afternoon >>> meetings at Berkeley that I'm sure Aron will recall. I don't think >>> date fields go back far enough in time, so I'd recommend using >>> integers or floating point numbers and providing the math >> in search. >>> It is also important to make sure that false precision isn't >>> introduced. If something is dated 1900, the system shouldn't enter >>> that as January 1, 1900 unless the uncertainty values can >> be used to >>> differentiate and the display date can be simply 1900. >>> >>> Susan >>> >>> >>> On 08/24/2011 09:16 AM, Patrick Schmitz wrote: >>>> Really? I am a little surprised that in all the design discussions >>>> for structured dates this did not come up. >>>> Anyway, I will say that we do not have the time to build >> some system >>>> that allows for a lot of custom behavior, or elaborate heuristics >>>> here (i.e., trying to divine what is intended from a set of text >>>> fields). I would strongly suggest that we either >>>> >>>> 1. >>>> require entry of some actual date fields upon which >> the application >>>> can reason, >>>> or >>>> 2. >>>> we define a very simple set of rules for building such >> date values >>>> from some of the existing text fields. By very simple, >> I mean that >>>> we only consider fields like year, month, day, and >> that the rules >>>> produce two scalar dates (early and late), which >> either define a >>>> range, or define a point (if they are equal). AIUI, a >> structured >>>> date /cannot/ model the idea of "some time/any time after (or >>>> before) a date". >>>> >>>> One further question is the semantic of which date (if >> any) will be >>>> used for ordering. >>>> If and when advanced search supports date ranges (which >> also implies >>>> support for /before/ and /after/ logic in search), we can >> handle the >>>> interval logic associated with the structured dates, but >> only if we >>>> have a pair of scalar date values. >>>> Down the road, I do think it would be interesting to >> implement some >>>> fuzzier logic on date search, to better support "circa" >> logic and the >>>> uncertainty values. However, I suspect that it will take longer to >>>> arrive at consensus on these semantics, and would defer that >>>> discussion for now. >>>> Thanks - Patrick >>>> >>>> >>>> >> -------------------------------------------------------------- >> ---------- >>>> *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu] >>>> *Sent:* Wednesday, August 24, 2011 9:01 AM >>>> *To:* Patrick Schmitz >>>> *Cc:* 'Christopher Pott'; 'Aron Roberts'; >>>> talk@lists.collectionspace.org >>>> *Subject:* Re: [Talk] what's the best way to represent >> structured >>>> dates? >>>> >>>> Hi Patrick, I think it's fair to say there's no >> consensus because >>>> there's been no discussion :-) It's possible that each >> customer will >>>> want to use structured dates differently, which will >> make it hard to >>>> add any intelligence or even predictability to things >> like search. >>>> However, I also think that if a significant number of deploying >>>> institutions can come to some consensus on best ways to use and >>>> search structured dates, then many institutions will >> choose to adopt >>>> the same best practices. >>>> >>>> Thanks, >>>> Chris >>>> >>>> On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote: >>>> >>>>> Aron was out yesterday, and Rick is out for a few >> days. They have >>>>> done a lot of the structured dates work to date. >>>>> The last I heard, Aron was hoping to get consensus >> from the users >>>>> about how we map all the text fields to actual date >> fields that we >>>>> can reason on (for search, ordering, etc). To date, >> there seems to >>>>> be no such consensus, which will preclude search, >> ordering, etc. >>>>> on structured dates. I would think that this would >> make structured >>>>> dates quite a bit less useful... >>>>> Perhaps (I hope) my understanding is out of date, and >> we just need >>>>> to implement the logic. Perhaps Aron can jump in this >> AM, when he >>>>> gets in. >>>>> Patrick >>>>> >>>>> >>>>> >> -------------------------------------------------------------- >> ---------- >>>>> *From:* talk-bounces@lists.collectionspace.org >>>>> <mailto:talk-bounces@lists.collectionspace.org> >>>>> [mailto:talk-bounces@lists.collectionspace.org] >> *On Behalf Of >>>>> *Chris Hoffman >>>>> *Sent:* Tuesday, August 23, 2011 11:05 AM >>>>> *To:* Christopher Pott >>>>> *Cc:* talk@lists.collectionspace.org >>>>> <mailto:talk@lists.collectionspace.org> >>>>> *Subject:* Re: [Talk] what's the best way to represent >>>>> structured dates? >>>>> >>>>> Hi Chris, >>>>> >>>>> Thanks for asking these questions. We've been >> wondering the >>>>> same things. Have you changed any existing calendar date >>>>> fields to structured dates or added new structured date >>>>> fields? I think this is all quite new, and >> together we might >>>>> be able to come up with some best practices >> (related to data >>>>> migration and data entry). >>>>> >>>>> I suspect that search and advanced search of >> structured date >>>>> fields is going to be an issue for those of us deploying >>>>> CollectionSpace. Our users will want CSpace to be >> smart about >>>>> structured dates, interpreting qualifiers, and so >> on, but I >>>>> bet that we won't see much as much as we would like in the >>>>> first version of advanced search. I'm always happy to be >>>>> proved wrong! :-) >>>>> >>>>> Chris >>>>> >>>>> On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm looking at the structured date schema and >> the best way to >>>>>> represent dates migrated from our old system. >> Any information >>>>>> about the following would be much appreciated.. >>>>>> >>>>>> 1. I see some different/overlapping possibilities for >>>>>> recording an object creation date. For example, >> to represent >>>>>> "1718" (display date) one could either store: >>>>>> >>>>>> a) a single date, providing year only (1718) >>>>>> >>>>>> b) an earliest date (01.01.1718) with a latest date >>>>>> (31.12.1718) >>>>>> >>>>>> c) a single date (01.01.1718) with a qualifier of + 364 >>>>>> days >>>>>> >>>>>> Is there any difference in the meaning of these >> three (and >>>>>> would they all be treated equally by services such as >>>>>> advanced search)? Our migration data is >> currently in format >>>>>> 'b' but the other formats are sometimes >> 'friendlier' for new >>>>>> input. >>>>>> >>>>>> 2. How are date "certainties" and "qualifiers" used by >>>>>> advanced search? >>>>>> >>>>>> I've specified "1718", as a creation date with a >> certainty of >>>>>> "circa". Will a search for objects created >> during 1717 result >>>>>> in a hit? >>>>>> >>>>>> Or is it necessary to use qualifiers (eg. 1718 >> +/- 2 years) >>>>>> to define exact limits for this type of searching? >>>>>> >>>>>> 3. Is the Date schema locally extendable? (We >> need a field to >>>>>> hold an English translation of the display text) >>>>>> >>>>>> 4. Will Advanced search by creation date also search the >>>>>> display date or will it just use the single/earliest and >>>>>> latest date groups? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Chris >>>>>> >>>>>> _______________________________________________ >>>>>> Talk mailing list >>>>>> Talk@lists.collectionspace.org >>>>>> <mailto:Talk@lists.collectionspace.org> >>>>>> >>>>>> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collec >>>>>> tionspace.org >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Talk mailing list >>>> Talk@lists.collectionspace.org >>>> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collecti >>>> onspace.org >>> >>> >>> _______________________________________________ >>> Talk mailing list >>> Talk@lists.collectionspace.org >>> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectio >>> nspace.org >>> >> >> _______________________________________________ >> Talk mailing list >> Talk@lists.collectionspace.org >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c >> ollectionspace.org >> >
MT
Michael T. Black
Wed, Aug 24, 2011 8:58 PM

Hi Patrick,

Just concluded a summit of the chronometric working group here at the Hearst (or, as we call it, me and Paolo), and we're happy with a 300Myr limit (BTW—we and presumably paleo, geology, etc. would call this BP, not BCE).  We have a handful of 50–150Myr BP objects for which we might conceivably have dates we'd want to enter.  However, we can't think of any cases in which we've got objects older than 300Myr for which the ≥ 300Myr date would have enough importance to make entry of these dates a requirement for us at the moment (e.g., we might have a bead made of an Ordovician brachiopod, but the collection and production dates would be the critical ones for us, not the date of the material from which the item was made).  I say "at the moment" because it would be awfully handy to have these dates entered once we're at the point where we can offer federated searching across collections/museums.

I can tell you, though, that you'll probably have a tough time convincing the paleo folks that they don't need to enter dates for their Carboniferous and earlier collections:

http://www.ucmp.berkeley.edu/science/inverts/carboniferous.php
http://www.ucmp.berkeley.edu/science/inverts/pennsylvanian.php
http://www.ucmp.berkeley.edu/science/inverts/mississippian.php
http://www.ucmp.berkeley.edu/science/inverts/devonian.php
http://www.ucmp.berkeley.edu/science/inverts/silurian.php
http://www.ucmp.berkeley.edu/science/inverts/ordovician.php
http://www.ucmp.berkeley.edu/science/inverts/cambrian.php
http://www.ucmp.berkeley.edu/science/inverts/precambrian.php

Michael

On Aug 24, 2011, at 12:00 PM, Patrick Schmitz wrote:

I think there is lots of potential for fuzzy date interpretation, and we can
pursue this as a second step.

Next, we need to find out if the 300M yr BCE range will work for Michael,
for Paleo, Essig, et al. I am worried about Paleo in particular.

Any other folks on this list with collections that go back a long way? How
much of a limitation would the 300M yr BCE range be?

Thanks - Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf Of
Aron Roberts
Sent: Wednesday, August 24, 2011 11:39 AM
To: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent structured dates?

On Wed, Aug 24, 2011 at 11:16 AM, Janice Eklund
janice.l.eklund@gmail.com wrote:

I see no problem with the scalar date scenarios listed below.

Thanks much, Patrick, for such a simple, sensible way of
creating single range start and end values, and Jan for your
feedback that this would work for you.

I've had many long discussions with Susan and Glen about
this topic and their frustrations with 'black box' or
otherwise unsatisfactory fuzzy date algorithms.  Using our
current model / schema for structured dates, with its
separate Year/Month/Day fields for range start/single date
and end date - for which they both get considerable credit -
and the clear logic you outline below, date range search can
be transparent and easy to understand.

Note that entry of Year/Month/Day values versus uncertainty
in the structure date popup is an either/or, as I understand
it.  If you want the transparent, easy-to-understand
behavior, you enter a year, and maybe also a month, or month
and day, for the start/single date, and
(optionally) for the end date.  If you enter uncertainty-related
values, when this is implemented down the road, perhaps a
pluggable fuzzy date handler would be invoked, a la Dick's suggestion?

Aron

On 8/24/11, Patrick Schmitz pschmitz@berkeley.edu wrote:

Maybe I am missing something, but is there a problem with

the following:

Single date, expressed as "1910". We model this (in earliest and
latest scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910"

Date range expressed as earliest:"1910", latest:"1915". We

model this

as
esd: "Jan 1 1910" and lsd: "Dec 31 1915"

Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We
model this as esd: "Jan 1 1910" and lsd: "Aug 31 1910"

The esd and lsd scalar dates are used for search and

ordering, not display.

Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf

Of Janice

Eklund
Sent: Wednesday, August 24, 2011 10:17 AM
To: Susan Stone
Cc: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured dates?

I second Susan's concern about false precision.  At least

90% of our

dates are year only single dates or date ranges. If you are
considering fuzzy logic down the road, look at the existing code
carefully.  The fuzzy date machine, at least as

implemented at UCB

for the HAVRC, does not handle ancient dates properly.

Jan

On 8/24/11, Susan Stone sstone@socrates.berkeley.edu wrote:

Patrick,

There was quite a bit of discussion of this in our

Wednesday afternoon

meetings at Berkeley that I'm sure Aron will recall. I

don't think

date fields go back far enough in time, so I'd recommend using
integers or floating point numbers and providing the math

in search.

It is also important to make sure that false precision isn't
introduced. If something is dated 1900, the system

shouldn't enter

that as January 1, 1900 unless the uncertainty values can

be used to

differentiate and the display date can be simply 1900.

Susan

On 08/24/2011 09:16 AM, Patrick Schmitz wrote:

Really? I am a little surprised that in all the design
discussions for structured dates this did not come up.
Anyway, I will say that we do not have the time to build

some system

that allows for a lot of custom behavior, or elaborate

heuristics

here (i.e., trying to divine what is intended from a

set of text

fields). I would strongly suggest that we either

  1. require entry of some actual date fields upon which

the application

 can reason,
 or
  1. we define a very simple set of rules for building such

date values

 from some of the existing text fields. By very simple,

I mean that

 we only consider fields like year, month, day, and

that the rules

 produce two scalar dates (early and late), which

either define a

 range, or define a point (if they are equal). AIUI, a

structured

 date /cannot/ model the idea of "some time/any 

time after (or

 before) a date".

One further question is the semantic of which date (if

any) will be

used for ordering.
If and when advanced search supports date ranges (which

also implies

support for /before/ and /after/ logic in search), we can

handle the

interval logic associated with the structured dates, but

only if we

have a pair of scalar date values.
Down the road, I do think it would be interesting to

implement some

fuzzier logic on date search, to better support "circa"

logic and the

uncertainty values. However, I suspect that it will

take longer

to arrive at consensus on these semantics, and would

defer that

discussion for now.
Thanks - Patrick



 *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
 *Sent:* Wednesday, August 24, 2011 9:01 AM
 *To:* Patrick Schmitz
 *Cc:* 'Christopher Pott'; 'Aron Roberts'; 

talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured

dates?

 Hi Patrick, I think it's fair to say there's no

consensus because

 there's been no discussion :-) It's possible that each

customer will

 want to use structured dates differently, which will

make it hard to

 add any intelligence or even predictability to things

like search.

 However, I also think that if a significant number of 

deploying
institutions can come to some consensus on best

ways to use

and
search structured dates, then many institutions will

choose to adopt

 the same best practices.

 Thanks,
 Chris

 On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote:
 Aron was out yesterday, and Rick is out for a few

days. They have

 done a lot of the structured dates work to date.
 The last I heard, Aron was hoping to get consensus

from the users

 about how we map all the text fields to actual date

fields that we

 can reason on (for search, ordering, etc). To date,

there seems to

 be no such consensus, which will preclude search,

ordering, etc.

 on structured dates. I would think that this would

make structured

 dates quite a bit less useful...
 Perhaps (I hope) my understanding is out of date, and

we just need

 to implement the logic. Perhaps Aron can jump in this

AM, when he

 gets in.
 Patrick


     *From:* talk-bounces@lists.collectionspace.org
     <mailto:talk-bounces@lists.collectionspace.org>
     [mailto:talk-bounces@lists.collectionspace.org]

*On Behalf Of

     *Chris Hoffman
     *Sent:* Tuesday, August 23, 2011 11:05 AM
     *To:* Christopher Pott
     *Cc:* talk@lists.collectionspace.org
     <mailto:talk@lists.collectionspace.org>
     *Subject:* Re: [Talk] what's the best way to represent
     structured dates?

     Hi Chris,

     Thanks for asking these questions. We've been

wondering the

     same things. Have you changed any existing 

calendar date

     fields to structured dates or added new 

structured date

     fields? I think this is all quite new, and

together we might

     be able to come up with some best practices

(related to data

     migration and data entry).

     I suspect that search and advanced search of

structured date

     fields is going to be an issue for those of 

us deploying

     CollectionSpace. Our users will want CSpace to be

smart about

     structured dates, interpreting qualifiers, and so

on, but I

     bet that we won't see much as much as we 

would like in

the
first version of advanced search. I'm always

happy to be

     proved wrong! :-)

     Chris

     On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote:
     Hi,

     I'm looking at the structured date schema and

the best way to

     represent dates migrated from our old system.

Any information

     about the following would be much appreciated..

     1. I see some different/overlapping possibilities for
     recording an object creation date. For example,

to represent

     "1718" (display date) one could either store:

     a) a single date, providing year only (1718)

     b) an earliest date (01.01.1718) with a latest date

(31.12.1718)

     c) a single date (01.01.1718) with a 

qualifier of + 364

days

     Is there any difference in the meaning of these

three (and

     would they all be treated equally by services such as
     advanced search)? Our migration data is

currently in format

     'b' but the other formats are sometimes

'friendlier' for new

     input.

     2. How are date "certainties" and 

"qualifiers" used by

     advanced search?

     I've specified "1718", as a creation date with a

certainty of

     "circa". Will a search for objects created

during 1717 result

     in a hit?

     Or is it necessary to use qualifiers (eg. 1718

+/- 2 years)

     to define exact limits for this type of searching?

     3. Is the Date schema locally extendable? (We

need a field to

     hold an English translation of the display text)

     4. Will Advanced search by creation date also search 

the
display date or will it just use the single/earliest
and
latest date groups?

     Thanks,

     Chris

     _______________________________________________
     Talk mailing list
     Talk@lists.collectionspace.org
     <mailto:Talk@lists.collectionspace.org>

tionspace.org


Talk mailing list
Talk@lists.collectionspace.org

i

onspace.org


Talk mailing list
Talk@lists.collectionspace.org

nspace.org

Hi Patrick, Just concluded a summit of the chronometric working group here at the Hearst (or, as we call it, me and Paolo), and we're happy with a 300Myr limit (BTW—we and presumably paleo, geology, etc. would call this BP, not BCE). We have a handful of 50–150Myr BP objects for which we might conceivably have dates we'd want to enter. However, we can't think of any cases in which we've got objects older than 300Myr for which the ≥ 300Myr date would have enough importance to make entry of these dates a requirement for us at the moment (e.g., we might have a bead made of an Ordovician brachiopod, but the collection and production dates would be the critical ones for us, not the date of the material from which the item was made). I say "at the moment" because it would be awfully handy to have these dates entered once we're at the point where we can offer federated searching across collections/museums. I can tell you, though, that you'll probably have a tough time convincing the paleo folks that they don't need to enter dates for their Carboniferous and earlier collections: http://www.ucmp.berkeley.edu/science/inverts/carboniferous.php http://www.ucmp.berkeley.edu/science/inverts/pennsylvanian.php http://www.ucmp.berkeley.edu/science/inverts/mississippian.php http://www.ucmp.berkeley.edu/science/inverts/devonian.php http://www.ucmp.berkeley.edu/science/inverts/silurian.php http://www.ucmp.berkeley.edu/science/inverts/ordovician.php http://www.ucmp.berkeley.edu/science/inverts/cambrian.php http://www.ucmp.berkeley.edu/science/inverts/precambrian.php Michael On Aug 24, 2011, at 12:00 PM, Patrick Schmitz wrote: > I think there is lots of potential for fuzzy date interpretation, and we can > pursue this as a second step. > > Next, we need to find out if the 300M yr BCE range will work for Michael, > for Paleo, Essig, et al. I am worried about Paleo in particular. > > Any other folks on this list with collections that go back a long way? How > much of a limitation would the 300M yr BCE range be? > > Thanks - Patrick > >> -----Original Message----- >> From: talk-bounces@lists.collectionspace.org >> [mailto:talk-bounces@lists.collectionspace.org] On Behalf Of >> Aron Roberts >> Sent: Wednesday, August 24, 2011 11:39 AM >> To: talk@lists.collectionspace.org >> Subject: Re: [Talk] what's the best way to represent structured dates? >> >> On Wed, Aug 24, 2011 at 11:16 AM, Janice Eklund >> <janice.l.eklund@gmail.com> wrote: >>> I see no problem with the scalar date scenarios listed below. >> >> Thanks much, Patrick, for such a simple, sensible way of >> creating single range start and end values, and Jan for your >> feedback that this would work for you. >> >> I've had many long discussions with Susan and Glen about >> this topic and their frustrations with 'black box' or >> otherwise unsatisfactory fuzzy date algorithms. Using our >> current model / schema for structured dates, with its >> separate Year/Month/Day fields for range start/single date >> and end date - for which they both get considerable credit - >> and the clear logic you outline below, date range search can >> be transparent and easy to understand. >> >> Note that entry of Year/Month/Day values versus uncertainty >> in the structure date popup is an either/or, as I understand >> it. If you want the transparent, easy-to-understand >> behavior, you enter a year, and maybe also a month, or month >> and day, for the start/single date, and >> (optionally) for the end date. If you enter uncertainty-related >> values, when this is implemented down the road, perhaps a >> pluggable fuzzy date handler would be invoked, a la Dick's suggestion? >> >> Aron >> >>> On 8/24/11, Patrick Schmitz <pschmitz@berkeley.edu> wrote: >>>> Maybe I am missing something, but is there a problem with >> the following: >>>> >>>> Single date, expressed as "1910". We model this (in earliest and >>>> latest scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910" >>>> >>>> Date range expressed as earliest:"1910", latest:"1915". We >> model this >>>> as >>>> esd: "Jan 1 1910" and lsd: "Dec 31 1915" >>>> >>>> Date range expressed as earliest:"Jan 1910", latest:"Aug 1910". We >>>> model this as esd: "Jan 1 1910" and lsd: "Aug 31 1910" >>>> >>>> The esd and lsd scalar dates are used for search and >> ordering, not display. >>>> >>>> Patrick >>>> >>>>> -----Original Message----- >>>>> From: talk-bounces@lists.collectionspace.org >>>>> [mailto:talk-bounces@lists.collectionspace.org] On Behalf >> Of Janice >>>>> Eklund >>>>> Sent: Wednesday, August 24, 2011 10:17 AM >>>>> To: Susan Stone >>>>> Cc: talk@lists.collectionspace.org >>>>> Subject: Re: [Talk] what's the best way to represent >> structured dates? >>>>> >>>>> I second Susan's concern about false precision. At least >> 90% of our >>>>> dates are year only single dates or date ranges. If you are >>>>> considering fuzzy logic down the road, look at the existing code >>>>> carefully. The fuzzy date machine, at least as >> implemented at UCB >>>>> for the HAVRC, does not handle ancient dates properly. >>>>> >>>>> Jan >>>>> >>>>> On 8/24/11, Susan Stone <sstone@socrates.berkeley.edu> wrote: >>>>>> Patrick, >>>>>> >>>>>> There was quite a bit of discussion of this in our >>>>> Wednesday afternoon >>>>>> meetings at Berkeley that I'm sure Aron will recall. I >> don't think >>>>>> date fields go back far enough in time, so I'd recommend using >>>>>> integers or floating point numbers and providing the math >>>>> in search. >>>>>> It is also important to make sure that false precision isn't >>>>>> introduced. If something is dated 1900, the system >> shouldn't enter >>>>>> that as January 1, 1900 unless the uncertainty values can >>>>> be used to >>>>>> differentiate and the display date can be simply 1900. >>>>>> >>>>>> Susan >>>>>> >>>>>> >>>>>> On 08/24/2011 09:16 AM, Patrick Schmitz wrote: >>>>>>> Really? I am a little surprised that in all the design >>>>>>> discussions for structured dates this did not come up. >>>>>>> Anyway, I will say that we do not have the time to build >>>>> some system >>>>>>> that allows for a lot of custom behavior, or elaborate >> heuristics >>>>>>> here (i.e., trying to divine what is intended from a >> set of text >>>>>>> fields). I would strongly suggest that we either >>>>>>> >>>>>>> 1. >>>>>>> require entry of some actual date fields upon which >>>>> the application >>>>>>> can reason, >>>>>>> or >>>>>>> 2. >>>>>>> we define a very simple set of rules for building such >>>>> date values >>>>>>> from some of the existing text fields. By very simple, >>>>> I mean that >>>>>>> we only consider fields like year, month, day, and >>>>> that the rules >>>>>>> produce two scalar dates (early and late), which >>>>> either define a >>>>>>> range, or define a point (if they are equal). AIUI, a >>>>> structured >>>>>>> date /cannot/ model the idea of "some time/any >> time after (or >>>>>>> before) a date". >>>>>>> >>>>>>> One further question is the semantic of which date (if >>>>> any) will be >>>>>>> used for ordering. >>>>>>> If and when advanced search supports date ranges (which >>>>> also implies >>>>>>> support for /before/ and /after/ logic in search), we can >>>>> handle the >>>>>>> interval logic associated with the structured dates, but >>>>> only if we >>>>>>> have a pair of scalar date values. >>>>>>> Down the road, I do think it would be interesting to >>>>> implement some >>>>>>> fuzzier logic on date search, to better support "circa" >>>>> logic and the >>>>>>> uncertainty values. However, I suspect that it will >> take longer >>>>>>> to arrive at consensus on these semantics, and would >> defer that >>>>>>> discussion for now. >>>>>>> Thanks - Patrick >>>>>>> >>>>>>> >>>>>>> >>>>> -------------------------------------------------------------- >>>>> ---------- >>>>>>> *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu] >>>>>>> *Sent:* Wednesday, August 24, 2011 9:01 AM >>>>>>> *To:* Patrick Schmitz >>>>>>> *Cc:* 'Christopher Pott'; 'Aron Roberts'; >>>>>>> talk@lists.collectionspace.org >>>>>>> *Subject:* Re: [Talk] what's the best way to represent >>>>> structured >>>>>>> dates? >>>>>>> >>>>>>> Hi Patrick, I think it's fair to say there's no >>>>> consensus because >>>>>>> there's been no discussion :-) It's possible that each >>>>> customer will >>>>>>> want to use structured dates differently, which will >>>>> make it hard to >>>>>>> add any intelligence or even predictability to things >>>>> like search. >>>>>>> However, I also think that if a significant number of >>>>>>> deploying >>>>>>> institutions can come to some consensus on best >> ways to use >>>>>>> and >>>>>>> search structured dates, then many institutions will >>>>> choose to adopt >>>>>>> the same best practices. >>>>>>> >>>>>>> Thanks, >>>>>>> Chris >>>>>>> >>>>>>> On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote: >>>>>>> >>>>>>>> Aron was out yesterday, and Rick is out for a few >>>>> days. They have >>>>>>>> done a lot of the structured dates work to date. >>>>>>>> The last I heard, Aron was hoping to get consensus >>>>> from the users >>>>>>>> about how we map all the text fields to actual date >>>>> fields that we >>>>>>>> can reason on (for search, ordering, etc). To date, >>>>> there seems to >>>>>>>> be no such consensus, which will preclude search, >>>>> ordering, etc. >>>>>>>> on structured dates. I would think that this would >>>>> make structured >>>>>>>> dates quite a bit less useful... >>>>>>>> Perhaps (I hope) my understanding is out of date, and >>>>> we just need >>>>>>>> to implement the logic. Perhaps Aron can jump in this >>>>> AM, when he >>>>>>>> gets in. >>>>>>>> Patrick >>>>>>>> >>>>>>>> >>>>>>>> >>>>> -------------------------------------------------------------- >>>>> ---------- >>>>>>>> *From:* talk-bounces@lists.collectionspace.org >>>>>>>> <mailto:talk-bounces@lists.collectionspace.org> >>>>>>>> [mailto:talk-bounces@lists.collectionspace.org] >>>>> *On Behalf Of >>>>>>>> *Chris Hoffman >>>>>>>> *Sent:* Tuesday, August 23, 2011 11:05 AM >>>>>>>> *To:* Christopher Pott >>>>>>>> *Cc:* talk@lists.collectionspace.org >>>>>>>> <mailto:talk@lists.collectionspace.org> >>>>>>>> *Subject:* Re: [Talk] what's the best way to represent >>>>>>>> structured dates? >>>>>>>> >>>>>>>> Hi Chris, >>>>>>>> >>>>>>>> Thanks for asking these questions. We've been >>>>> wondering the >>>>>>>> same things. Have you changed any existing >> calendar date >>>>>>>> fields to structured dates or added new >> structured date >>>>>>>> fields? I think this is all quite new, and >>>>> together we might >>>>>>>> be able to come up with some best practices >>>>> (related to data >>>>>>>> migration and data entry). >>>>>>>> >>>>>>>> I suspect that search and advanced search of >>>>> structured date >>>>>>>> fields is going to be an issue for those of >> us deploying >>>>>>>> CollectionSpace. Our users will want CSpace to be >>>>> smart about >>>>>>>> structured dates, interpreting qualifiers, and so >>>>> on, but I >>>>>>>> bet that we won't see much as much as we >> would like in >>>>>>>> the >>>>>>>> first version of advanced search. I'm always >> happy to be >>>>>>>> proved wrong! :-) >>>>>>>> >>>>>>>> Chris >>>>>>>> >>>>>>>> On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I'm looking at the structured date schema and >>>>> the best way to >>>>>>>>> represent dates migrated from our old system. >>>>> Any information >>>>>>>>> about the following would be much appreciated.. >>>>>>>>> >>>>>>>>> 1. I see some different/overlapping possibilities for >>>>>>>>> recording an object creation date. For example, >>>>> to represent >>>>>>>>> "1718" (display date) one could either store: >>>>>>>>> >>>>>>>>> a) a single date, providing year only (1718) >>>>>>>>> >>>>>>>>> b) an earliest date (01.01.1718) with a latest date >>>>>>>>> (31.12.1718) >>>>>>>>> >>>>>>>>> c) a single date (01.01.1718) with a >> qualifier of + 364 >>>>>>>>> days >>>>>>>>> >>>>>>>>> Is there any difference in the meaning of these >>>>> three (and >>>>>>>>> would they all be treated equally by services such as >>>>>>>>> advanced search)? Our migration data is >>>>> currently in format >>>>>>>>> 'b' but the other formats are sometimes >>>>> 'friendlier' for new >>>>>>>>> input. >>>>>>>>> >>>>>>>>> 2. How are date "certainties" and >> "qualifiers" used by >>>>>>>>> advanced search? >>>>>>>>> >>>>>>>>> I've specified "1718", as a creation date with a >>>>> certainty of >>>>>>>>> "circa". Will a search for objects created >>>>> during 1717 result >>>>>>>>> in a hit? >>>>>>>>> >>>>>>>>> Or is it necessary to use qualifiers (eg. 1718 >>>>> +/- 2 years) >>>>>>>>> to define exact limits for this type of searching? >>>>>>>>> >>>>>>>>> 3. Is the Date schema locally extendable? (We >>>>> need a field to >>>>>>>>> hold an English translation of the display text) >>>>>>>>> >>>>>>>>> 4. Will Advanced search by creation date also search >>>>>>>>> the >>>>>>>>> display date or will it just use the single/earliest >>>>>>>>> and >>>>>>>>> latest date groups? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Chris >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Talk mailing list >>>>>>>>> Talk@lists.collectionspace.org >>>>>>>>> <mailto:Talk@lists.collectionspace.org> >>>>>>>>> >>>>>>>>> >>>>> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collec >>>>>>>>> tionspace.org >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Talk mailing list >>>>>>> Talk@lists.collectionspace.org >>>>>>> >>>>> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collect >>>>> i >>>>>>> onspace.org >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Talk mailing list >>>>>> Talk@lists.collectionspace.org >>>>>> >>>>> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collect >>>>> io >>>>>> nspace.org >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Talk mailing list >>>>> Talk@lists.collectionspace.org >>>>> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c >>>>> ollectionspace.org >>>>> >>>> >>>> >>> >>> _______________________________________________ >>> Talk mailing list >>> Talk@lists.collectionspace.org >>> >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectio >>> nspace.org >>> >> >> _______________________________________________ >> Talk mailing list >> Talk@lists.collectionspace.org >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c >> ollectionspace.org >> >
PS
Patrick Schmitz
Wed, Aug 24, 2011 9:46 PM

So this is what I more or less expected.

I see several alternatives:

  1. We do something special for Paleo
  2. We implement our own Calendar support that can get more range out of a
    Long storage unit.

2 would be a lot more work, and I am not thrilled with that approach.

I am pretty sure I do not know enough about how Paleo and friends enter
dates, interpret dates, compare them, etc., etc. If they actually use a
different model for these dates, then perhaps 1 is a reasonable approach.
E.g., we could support a different scalar date field that used a different
unit (like years, rather than days), and then they could do searches on that
rather than date fields. A long range of years into the past (E.g., rooted
at 3000 CE) could represent years nearly back to the beginning of our planet
(about 4.2 billion years ago). While astronomic time has a larger range, we
can cross that bridge with a similar approach (e.g., an extension field that
represents decades).

Patrick

-----Original Message-----
From: Michael T. Black [mailto:mtblack@berkeley.edu]
Sent: Wednesday, August 24, 2011 1:59 PM
To: Patrick Schmitz
Cc: Aron Roberts; talk@lists.collectionspace.org; Paolo Pellegatti
Subject: Re: [Talk] what's the best way to represent structured dates?

Hi Patrick,

Just concluded a summit of the chronometric working 

group here at the Hearst (or, as we call it, me and Paolo),
and we're happy with a 300Myr limit (BTW-we and presumably
paleo, geology, etc. would call this BP, not BCE).  We have a
handful of 50-150Myr BP objects for which we might
conceivably have dates we'd want to enter.  However, we can't
think of any cases in which we've got objects older than
300Myr for which the = 300Myr date would have enough
importance to make entry of these dates a requirement for us
at the moment (e.g., we might have a bead made of an
Ordovician brachiopod, but the collection and production
dates would be the critical ones for us, not the date of the
material from which the item was made).  I say "at the
moment" because it would be awfully handy to have these dates
entered once we're at the point where we can offer federated
searching across collections/museums.

I can tell you, though, that you'll probably have a 

tough time convincing the paleo folks that they don't need to
enter dates for their Carboniferous and earlier collections:

http://www.ucmp.berkeley.edu/science/inverts/carboniferous.php
http://www.ucmp.berkeley.edu/science/inverts/pennsylvanian.php
http://www.ucmp.berkeley.edu/science/inverts/mississippian.php
http://www.ucmp.berkeley.edu/science/inverts/devonian.php
http://www.ucmp.berkeley.edu/science/inverts/silurian.php
http://www.ucmp.berkeley.edu/science/inverts/ordovician.php
http://www.ucmp.berkeley.edu/science/inverts/cambrian.php
http://www.ucmp.berkeley.edu/science/inverts/precambrian.php

Michael

On Aug 24, 2011, at 12:00 PM, Patrick Schmitz wrote:

I think there is lots of potential for fuzzy date

interpretation, and

we can pursue this as a second step.

Next, we need to find out if the 300M yr BCE range will work for
Michael, for Paleo, Essig, et al. I am worried about Paleo

in particular.

Any other folks on this list with collections that go back

a long way?

How much of a limitation would the 300M yr BCE range be?

Thanks - Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf Of Aron
Roberts
Sent: Wednesday, August 24, 2011 11:39 AM
To: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured dates?

On Wed, Aug 24, 2011 at 11:16 AM, Janice Eklund
janice.l.eklund@gmail.com wrote:

I see no problem with the scalar date scenarios listed below.

Thanks much, Patrick, for such a simple, sensible way of creating
single range start and end values, and Jan for your feedback that
this would work for you.

I've had many long discussions with Susan and Glen about

this topic

and their frustrations with 'black box' or otherwise

unsatisfactory

fuzzy date algorithms.  Using our current model / schema for
structured dates, with its separate Year/Month/Day fields

for range

start/single date and end date - for which they both get

considerable

credit - and the clear logic you outline below, date range

search can

be transparent and easy to understand.

Note that entry of Year/Month/Day values versus

uncertainty in the

structure date popup is an either/or, as I understand it.  If you
want the transparent, easy-to-understand behavior, you

enter a year,

and maybe also a month, or month and day, for the

start/single date,

and
(optionally) for the end date.  If you enter uncertainty-related
values, when this is implemented down the road, perhaps a

pluggable

fuzzy date handler would be invoked, a la Dick's suggestion?

Aron

On 8/24/11, Patrick Schmitz pschmitz@berkeley.edu wrote:

Maybe I am missing something, but is there a problem with

the following:

Single date, expressed as "1910". We model this (in earliest and
latest scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910"

Date range expressed as earliest:"1910", latest:"1915". We

model this

as
esd: "Jan 1 1910" and lsd: "Dec 31 1915"

Date range expressed as earliest:"Jan 1910", latest:"Aug

1910". We

model this as esd: "Jan 1 1910" and lsd: "Aug 31 1910"

The esd and lsd scalar dates are used for search and

ordering, not display.

Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf

Of Janice

Eklund
Sent: Wednesday, August 24, 2011 10:17 AM
To: Susan Stone
Cc: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured dates?

I second Susan's concern about false precision.  At least

90% of our

dates are year only single dates or date ranges. If you are
considering fuzzy logic down the road, look at the

existing code

carefully.  The fuzzy date machine, at least as

implemented at UCB

for the HAVRC, does not handle ancient dates properly.

Jan

On 8/24/11, Susan Stone sstone@socrates.berkeley.edu wrote:

Patrick,

There was quite a bit of discussion of this in our

Wednesday afternoon

meetings at Berkeley that I'm sure Aron will recall. I

don't think

date fields go back far enough in time, so I'd recommend using
integers or floating point numbers and providing the math

in search.

It is also important to make sure that false precision isn't
introduced. If something is dated 1900, the system

shouldn't enter

that as January 1, 1900 unless the uncertainty values can

be used to

differentiate and the display date can be simply 1900.

Susan

On 08/24/2011 09:16 AM, Patrick Schmitz wrote:

Really? I am a little surprised that in all the design
discussions for structured dates this did not come up.
Anyway, I will say that we do not have the time to build

some system

that allows for a lot of custom behavior, or elaborate

heuristics

here (i.e., trying to divine what is intended from a

set of text

fields). I would strongly suggest that we either

  1. require entry of some actual date fields upon which

the application

 can reason,
 or
  1. we define a very simple set of rules for building such

date values

 from some of the existing text fields. By very simple,

I mean that

 we only consider fields like year, month, day, and

that the rules

 produce two scalar dates (early and late), which

either define a

 range, or define a point (if they are equal). AIUI, a

structured

 date /cannot/ model the idea of "some time/any

time after (or

 before) a date".

One further question is the semantic of which date (if

any) will be

used for ordering.
If and when advanced search supports date ranges (which

also implies

support for /before/ and /after/ logic in search), we can

handle the

interval logic associated with the structured dates, but

only if we

have a pair of scalar date values.
Down the road, I do think it would be interesting to

implement some

fuzzier logic on date search, to better support "circa"

logic and the

uncertainty values. However, I suspect that it will

take longer

to arrive at consensus on these semantics, and would

defer that

discussion for now.
Thanks - Patrick



 *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
 *Sent:* Wednesday, August 24, 2011 9:01 AM
 *To:* Patrick Schmitz
 *Cc:* 'Christopher Pott'; 'Aron Roberts'; 

talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured

dates?

 Hi Patrick, I think it's fair to say there's no

consensus because

 there's been no discussion :-) It's possible that each

customer will

 want to use structured dates differently, which will

make it hard to

 add any intelligence or even predictability to things

like search.

 However, I also think that if a significant number of 

deploying
institutions can come to some consensus on best

ways to use

and
search structured dates, then many institutions will

choose to adopt

 the same best practices.

 Thanks,
 Chris

 On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote:
 Aron was out yesterday, and Rick is out for a few

days. They have

 done a lot of the structured dates work to date.
 The last I heard, Aron was hoping to get consensus

from the users

 about how we map all the text fields to actual date

fields that we

 can reason on (for search, ordering, etc). To date,

there seems to

 be no such consensus, which will preclude search,

ordering, etc.

 on structured dates. I would think that this would

make structured

 dates quite a bit less useful...
 Perhaps (I hope) my understanding is out of date, and

we just need

 to implement the logic. Perhaps Aron can jump in this

AM, when he

 gets in.
 Patrick


     *From:* talk-bounces@lists.collectionspace.org
     <mailto:talk-bounces@lists.collectionspace.org>
     [mailto:talk-bounces@lists.collectionspace.org]

*On Behalf Of

     *Chris Hoffman
     *Sent:* Tuesday, August 23, 2011 11:05 AM
     *To:* Christopher Pott
     *Cc:* talk@lists.collectionspace.org
     <mailto:talk@lists.collectionspace.org>
     *Subject:* Re: [Talk] what's the best way to 

represent

     structured dates?

     Hi Chris,

     Thanks for asking these questions. We've been

wondering the

     same things. Have you changed any existing

calendar date

     fields to structured dates or added new

structured date

     fields? I think this is all quite new, and

together we might

     be able to come up with some best practices

(related to data

     migration and data entry).

     I suspect that search and advanced search of

structured date

     fields is going to be an issue for those of

us deploying

     CollectionSpace. Our users will want CSpace to be

smart about

     structured dates, interpreting qualifiers, and so

on, but I

     bet that we won't see much as much as we

would like in

the
first version of advanced search. I'm always

happy to be

     proved wrong! :-)

     Chris

     On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote:
     Hi,

     I'm looking at the structured date schema and

the best way to

     represent dates migrated from our old system.

Any information

     about the following would be much appreciated..

     1. I see some different/overlapping 

possibilities for

     recording an object creation date. For example,

to represent

     "1718" (display date) one could either store:

     a) a single date, providing year only (1718)

     b) an earliest date (01.01.1718) with a latest date

(31.12.1718)

     c) a single date (01.01.1718) with a

qualifier of + 364

days

     Is there any difference in the meaning of these

three (and

     would they all be treated equally by 

services such as

     advanced search)? Our migration data is

currently in format

     'b' but the other formats are sometimes

'friendlier' for new

     input.

     2. How are date "certainties" and

"qualifiers" used by

     advanced search?

     I've specified "1718", as a creation date with a

certainty of

     "circa". Will a search for objects created

during 1717 result

     in a hit?

     Or is it necessary to use qualifiers (eg. 1718

+/- 2 years)

     to define exact limits for this type of searching?

     3. Is the Date schema locally extendable? (We

need a field to

     hold an English translation of the display text)

     4. Will Advanced search by creation date 

also search

the
display date or will it just use the

single/earliest

and
latest date groups?

     Thanks,

     Chris

     _______________________________________________
     Talk mailing list
     Talk@lists.collectionspace.org
     <mailto:Talk@lists.collectionspace.org>

tionspace.org


Talk mailing list
Talk@lists.collectionspace.org

i

onspace.org


Talk mailing list
Talk@lists.collectionspace.org

So this is what I more or less expected. I see several alternatives: 1) We do something special for Paleo 2) We implement our own Calendar support that can get more range out of a Long storage unit. 2 would be a lot more work, and I am not thrilled with that approach. I am pretty sure I do not know enough about how Paleo and friends enter dates, interpret dates, compare them, etc., etc. If they actually use a different model for these dates, then perhaps 1 is a reasonable approach. E.g., we could support a different scalar date field that used a different unit (like years, rather than days), and then they could do searches on that rather than date fields. A long range of years into the past (E.g., rooted at 3000 CE) could represent years nearly back to the beginning of our planet (about 4.2 billion years ago). While astronomic time has a larger range, we can cross that bridge with a similar approach (e.g., an extension field that represents decades). Patrick > -----Original Message----- > From: Michael T. Black [mailto:mtblack@berkeley.edu] > Sent: Wednesday, August 24, 2011 1:59 PM > To: Patrick Schmitz > Cc: Aron Roberts; talk@lists.collectionspace.org; Paolo Pellegatti > Subject: Re: [Talk] what's the best way to represent structured dates? > > Hi Patrick, > > Just concluded a summit of the chronometric working > group here at the Hearst (or, as we call it, me and Paolo), > and we're happy with a 300Myr limit (BTW-we and presumably > paleo, geology, etc. would call this BP, not BCE). We have a > handful of 50-150Myr BP objects for which we might > conceivably have dates we'd want to enter. However, we can't > think of any cases in which we've got objects older than > 300Myr for which the = 300Myr date would have enough > importance to make entry of these dates a requirement for us > at the moment (e.g., we might have a bead made of an > Ordovician brachiopod, but the collection and production > dates would be the critical ones for us, not the date of the > material from which the item was made). I say "at the > moment" because it would be awfully handy to have these dates > entered once we're at the point where we can offer federated > searching across collections/museums. > > I can tell you, though, that you'll probably have a > tough time convincing the paleo folks that they don't need to > enter dates for their Carboniferous and earlier collections: > > http://www.ucmp.berkeley.edu/science/inverts/carboniferous.php > http://www.ucmp.berkeley.edu/science/inverts/pennsylvanian.php > http://www.ucmp.berkeley.edu/science/inverts/mississippian.php > http://www.ucmp.berkeley.edu/science/inverts/devonian.php > http://www.ucmp.berkeley.edu/science/inverts/silurian.php > http://www.ucmp.berkeley.edu/science/inverts/ordovician.php > http://www.ucmp.berkeley.edu/science/inverts/cambrian.php > http://www.ucmp.berkeley.edu/science/inverts/precambrian.php > > Michael > > On Aug 24, 2011, at 12:00 PM, Patrick Schmitz wrote: > > > I think there is lots of potential for fuzzy date > interpretation, and > > we can pursue this as a second step. > > > > Next, we need to find out if the 300M yr BCE range will work for > > Michael, for Paleo, Essig, et al. I am worried about Paleo > in particular. > > > > Any other folks on this list with collections that go back > a long way? > > How much of a limitation would the 300M yr BCE range be? > > > > Thanks - Patrick > > > >> -----Original Message----- > >> From: talk-bounces@lists.collectionspace.org > >> [mailto:talk-bounces@lists.collectionspace.org] On Behalf Of Aron > >> Roberts > >> Sent: Wednesday, August 24, 2011 11:39 AM > >> To: talk@lists.collectionspace.org > >> Subject: Re: [Talk] what's the best way to represent > structured dates? > >> > >> On Wed, Aug 24, 2011 at 11:16 AM, Janice Eklund > >> <janice.l.eklund@gmail.com> wrote: > >>> I see no problem with the scalar date scenarios listed below. > >> > >> Thanks much, Patrick, for such a simple, sensible way of creating > >> single range start and end values, and Jan for your feedback that > >> this would work for you. > >> > >> I've had many long discussions with Susan and Glen about > this topic > >> and their frustrations with 'black box' or otherwise > unsatisfactory > >> fuzzy date algorithms. Using our current model / schema for > >> structured dates, with its separate Year/Month/Day fields > for range > >> start/single date and end date - for which they both get > considerable > >> credit - and the clear logic you outline below, date range > search can > >> be transparent and easy to understand. > >> > >> Note that entry of Year/Month/Day values versus > uncertainty in the > >> structure date popup is an either/or, as I understand it. If you > >> want the transparent, easy-to-understand behavior, you > enter a year, > >> and maybe also a month, or month and day, for the > start/single date, > >> and > >> (optionally) for the end date. If you enter uncertainty-related > >> values, when this is implemented down the road, perhaps a > pluggable > >> fuzzy date handler would be invoked, a la Dick's suggestion? > >> > >> Aron > >> > >>> On 8/24/11, Patrick Schmitz <pschmitz@berkeley.edu> wrote: > >>>> Maybe I am missing something, but is there a problem with > >> the following: > >>>> > >>>> Single date, expressed as "1910". We model this (in earliest and > >>>> latest scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910" > >>>> > >>>> Date range expressed as earliest:"1910", latest:"1915". We > >> model this > >>>> as > >>>> esd: "Jan 1 1910" and lsd: "Dec 31 1915" > >>>> > >>>> Date range expressed as earliest:"Jan 1910", latest:"Aug > 1910". We > >>>> model this as esd: "Jan 1 1910" and lsd: "Aug 31 1910" > >>>> > >>>> The esd and lsd scalar dates are used for search and > >> ordering, not display. > >>>> > >>>> Patrick > >>>> > >>>>> -----Original Message----- > >>>>> From: talk-bounces@lists.collectionspace.org > >>>>> [mailto:talk-bounces@lists.collectionspace.org] On Behalf > >> Of Janice > >>>>> Eklund > >>>>> Sent: Wednesday, August 24, 2011 10:17 AM > >>>>> To: Susan Stone > >>>>> Cc: talk@lists.collectionspace.org > >>>>> Subject: Re: [Talk] what's the best way to represent > >> structured dates? > >>>>> > >>>>> I second Susan's concern about false precision. At least > >> 90% of our > >>>>> dates are year only single dates or date ranges. If you are > >>>>> considering fuzzy logic down the road, look at the > existing code > >>>>> carefully. The fuzzy date machine, at least as > >> implemented at UCB > >>>>> for the HAVRC, does not handle ancient dates properly. > >>>>> > >>>>> Jan > >>>>> > >>>>> On 8/24/11, Susan Stone <sstone@socrates.berkeley.edu> wrote: > >>>>>> Patrick, > >>>>>> > >>>>>> There was quite a bit of discussion of this in our > >>>>> Wednesday afternoon > >>>>>> meetings at Berkeley that I'm sure Aron will recall. I > >> don't think > >>>>>> date fields go back far enough in time, so I'd recommend using > >>>>>> integers or floating point numbers and providing the math > >>>>> in search. > >>>>>> It is also important to make sure that false precision isn't > >>>>>> introduced. If something is dated 1900, the system > >> shouldn't enter > >>>>>> that as January 1, 1900 unless the uncertainty values can > >>>>> be used to > >>>>>> differentiate and the display date can be simply 1900. > >>>>>> > >>>>>> Susan > >>>>>> > >>>>>> > >>>>>> On 08/24/2011 09:16 AM, Patrick Schmitz wrote: > >>>>>>> Really? I am a little surprised that in all the design > >>>>>>> discussions for structured dates this did not come up. > >>>>>>> Anyway, I will say that we do not have the time to build > >>>>> some system > >>>>>>> that allows for a lot of custom behavior, or elaborate > >> heuristics > >>>>>>> here (i.e., trying to divine what is intended from a > >> set of text > >>>>>>> fields). I would strongly suggest that we either > >>>>>>> > >>>>>>> 1. > >>>>>>> require entry of some actual date fields upon which > >>>>> the application > >>>>>>> can reason, > >>>>>>> or > >>>>>>> 2. > >>>>>>> we define a very simple set of rules for building such > >>>>> date values > >>>>>>> from some of the existing text fields. By very simple, > >>>>> I mean that > >>>>>>> we only consider fields like year, month, day, and > >>>>> that the rules > >>>>>>> produce two scalar dates (early and late), which > >>>>> either define a > >>>>>>> range, or define a point (if they are equal). AIUI, a > >>>>> structured > >>>>>>> date /cannot/ model the idea of "some time/any > >> time after (or > >>>>>>> before) a date". > >>>>>>> > >>>>>>> One further question is the semantic of which date (if > >>>>> any) will be > >>>>>>> used for ordering. > >>>>>>> If and when advanced search supports date ranges (which > >>>>> also implies > >>>>>>> support for /before/ and /after/ logic in search), we can > >>>>> handle the > >>>>>>> interval logic associated with the structured dates, but > >>>>> only if we > >>>>>>> have a pair of scalar date values. > >>>>>>> Down the road, I do think it would be interesting to > >>>>> implement some > >>>>>>> fuzzier logic on date search, to better support "circa" > >>>>> logic and the > >>>>>>> uncertainty values. However, I suspect that it will > >> take longer > >>>>>>> to arrive at consensus on these semantics, and would > >> defer that > >>>>>>> discussion for now. > >>>>>>> Thanks - Patrick > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> -------------------------------------------------------------- > >>>>> ---------- > >>>>>>> *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu] > >>>>>>> *Sent:* Wednesday, August 24, 2011 9:01 AM > >>>>>>> *To:* Patrick Schmitz > >>>>>>> *Cc:* 'Christopher Pott'; 'Aron Roberts'; > >>>>>>> talk@lists.collectionspace.org > >>>>>>> *Subject:* Re: [Talk] what's the best way to represent > >>>>> structured > >>>>>>> dates? > >>>>>>> > >>>>>>> Hi Patrick, I think it's fair to say there's no > >>>>> consensus because > >>>>>>> there's been no discussion :-) It's possible that each > >>>>> customer will > >>>>>>> want to use structured dates differently, which will > >>>>> make it hard to > >>>>>>> add any intelligence or even predictability to things > >>>>> like search. > >>>>>>> However, I also think that if a significant number of > >>>>>>> deploying > >>>>>>> institutions can come to some consensus on best > >> ways to use > >>>>>>> and > >>>>>>> search structured dates, then many institutions will > >>>>> choose to adopt > >>>>>>> the same best practices. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Chris > >>>>>>> > >>>>>>> On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote: > >>>>>>> > >>>>>>>> Aron was out yesterday, and Rick is out for a few > >>>>> days. They have > >>>>>>>> done a lot of the structured dates work to date. > >>>>>>>> The last I heard, Aron was hoping to get consensus > >>>>> from the users > >>>>>>>> about how we map all the text fields to actual date > >>>>> fields that we > >>>>>>>> can reason on (for search, ordering, etc). To date, > >>>>> there seems to > >>>>>>>> be no such consensus, which will preclude search, > >>>>> ordering, etc. > >>>>>>>> on structured dates. I would think that this would > >>>>> make structured > >>>>>>>> dates quite a bit less useful... > >>>>>>>> Perhaps (I hope) my understanding is out of date, and > >>>>> we just need > >>>>>>>> to implement the logic. Perhaps Aron can jump in this > >>>>> AM, when he > >>>>>>>> gets in. > >>>>>>>> Patrick > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>> -------------------------------------------------------------- > >>>>> ---------- > >>>>>>>> *From:* talk-bounces@lists.collectionspace.org > >>>>>>>> <mailto:talk-bounces@lists.collectionspace.org> > >>>>>>>> [mailto:talk-bounces@lists.collectionspace.org] > >>>>> *On Behalf Of > >>>>>>>> *Chris Hoffman > >>>>>>>> *Sent:* Tuesday, August 23, 2011 11:05 AM > >>>>>>>> *To:* Christopher Pott > >>>>>>>> *Cc:* talk@lists.collectionspace.org > >>>>>>>> <mailto:talk@lists.collectionspace.org> > >>>>>>>> *Subject:* Re: [Talk] what's the best way to > represent > >>>>>>>> structured dates? > >>>>>>>> > >>>>>>>> Hi Chris, > >>>>>>>> > >>>>>>>> Thanks for asking these questions. We've been > >>>>> wondering the > >>>>>>>> same things. Have you changed any existing > >> calendar date > >>>>>>>> fields to structured dates or added new > >> structured date > >>>>>>>> fields? I think this is all quite new, and > >>>>> together we might > >>>>>>>> be able to come up with some best practices > >>>>> (related to data > >>>>>>>> migration and data entry). > >>>>>>>> > >>>>>>>> I suspect that search and advanced search of > >>>>> structured date > >>>>>>>> fields is going to be an issue for those of > >> us deploying > >>>>>>>> CollectionSpace. Our users will want CSpace to be > >>>>> smart about > >>>>>>>> structured dates, interpreting qualifiers, and so > >>>>> on, but I > >>>>>>>> bet that we won't see much as much as we > >> would like in > >>>>>>>> the > >>>>>>>> first version of advanced search. I'm always > >> happy to be > >>>>>>>> proved wrong! :-) > >>>>>>>> > >>>>>>>> Chris > >>>>>>>> > >>>>>>>> On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote: > >>>>>>>> > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> I'm looking at the structured date schema and > >>>>> the best way to > >>>>>>>>> represent dates migrated from our old system. > >>>>> Any information > >>>>>>>>> about the following would be much appreciated.. > >>>>>>>>> > >>>>>>>>> 1. I see some different/overlapping > possibilities for > >>>>>>>>> recording an object creation date. For example, > >>>>> to represent > >>>>>>>>> "1718" (display date) one could either store: > >>>>>>>>> > >>>>>>>>> a) a single date, providing year only (1718) > >>>>>>>>> > >>>>>>>>> b) an earliest date (01.01.1718) with a latest date > >>>>>>>>> (31.12.1718) > >>>>>>>>> > >>>>>>>>> c) a single date (01.01.1718) with a > >> qualifier of + 364 > >>>>>>>>> days > >>>>>>>>> > >>>>>>>>> Is there any difference in the meaning of these > >>>>> three (and > >>>>>>>>> would they all be treated equally by > services such as > >>>>>>>>> advanced search)? Our migration data is > >>>>> currently in format > >>>>>>>>> 'b' but the other formats are sometimes > >>>>> 'friendlier' for new > >>>>>>>>> input. > >>>>>>>>> > >>>>>>>>> 2. How are date "certainties" and > >> "qualifiers" used by > >>>>>>>>> advanced search? > >>>>>>>>> > >>>>>>>>> I've specified "1718", as a creation date with a > >>>>> certainty of > >>>>>>>>> "circa". Will a search for objects created > >>>>> during 1717 result > >>>>>>>>> in a hit? > >>>>>>>>> > >>>>>>>>> Or is it necessary to use qualifiers (eg. 1718 > >>>>> +/- 2 years) > >>>>>>>>> to define exact limits for this type of searching? > >>>>>>>>> > >>>>>>>>> 3. Is the Date schema locally extendable? (We > >>>>> need a field to > >>>>>>>>> hold an English translation of the display text) > >>>>>>>>> > >>>>>>>>> 4. Will Advanced search by creation date > also search > >>>>>>>>> the > >>>>>>>>> display date or will it just use the > single/earliest > >>>>>>>>> and > >>>>>>>>> latest date groups? > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> > >>>>>>>>> Chris > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> Talk mailing list > >>>>>>>>> Talk@lists.collectionspace.org > >>>>>>>>> <mailto:Talk@lists.collectionspace.org> > >>>>>>>>> > >>>>>>>>> > >>>>> > >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collec > >>>>>>>>> tionspace.org > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> Talk mailing list > >>>>>>> Talk@lists.collectionspace.org > >>>>>>> > >>>>> > >> > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collect > >>>>> i > >>>>>>> onspace.org > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> Talk mailing list > >>>>>> Talk@lists.collectionspace.org > >>>>>> > >>>>> > >> > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collect > >>>>> io > >>>>>> nspace.org > >>>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> Talk mailing list > >>>>> Talk@lists.collectionspace.org > >>>>> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c > >>>>> ollectionspace.org > >>>>> > >>>> > >>>> > >>> > >>> _______________________________________________ > >>> Talk mailing list > >>> Talk@lists.collectionspace.org > >>> > >> > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collecti > >> o > >>> nspace.org > >>> > >> > >> _______________________________________________ > >> Talk mailing list > >> Talk@lists.collectionspace.org > >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c > >> ollectionspace.org > >> > > > >
CP
Christopher Pott
Thu, Aug 25, 2011 12:13 PM

Thanks for the responses. Can you tell me if these interpretations of your responses are correct....

  • if we always provide a full (dd-MM-yyyy GG) earliest and latest date, these will be the dates used by services for search and ordering
  • if we provide less information (such as just a year in earliest date) services will generate an internal earliest and latest date to cover that period
  • Services won't be parsing display dates to generate earliest and latest dates
  • Services won't be examining  'qualifiers' or 'certainty' fields so if we rely on these then searching will fail

Is the date schema extensible? We need a way to store some information with a date, which probably wouldn't be useful to services but that can help us modify the UI to our requirements.

Our earliest structured date is currently 1600 BC which I thought was quite old until I read the other posts.

Just out of interest to other implementers, some screen shots of our planned date UI implementation can be seen here http://wiki.collectionspace.org/display/collectionspace/Date+and+Time+Use+Cases. It will use a combination of drop down lists and day/month/year inputs to enable generation of display dates according to some precise rules (required for catalogues and signs). It should generate a translation at the same time, as well as the earliest and latest dates.

Cheers,
Chris

-----Oprindelig meddelelse-----
Fra: talk-bounces@lists.collectionspace.org [mailto:talk-
bounces@lists.collectionspace.org] På vegne af Patrick Schmitz
Sendt: 24. august 2011 23:47
Til: 'Michael T. Black'
Cc: talk@lists.collectionspace.org
Emne: Re: [Talk] what's the best way to represent structured dates?

So this is what I more or less expected.

I see several alternatives:

  1. We do something special for Paleo
  2. We implement our own Calendar support that can get more range out of a
    Long storage unit.

2 would be a lot more work, and I am not thrilled with that approach.

I am pretty sure I do not know enough about how Paleo and friends enter
dates, interpret dates, compare them, etc., etc. If they actually use a
different model for these dates, then perhaps 1 is a reasonable approach.
E.g., we could support a different scalar date field that used a different
unit (like years, rather than days), and then they could do searches on
that
rather than date fields. A long range of years into the past (E.g., rooted
at 3000 CE) could represent years nearly back to the beginning of our
planet
(about 4.2 billion years ago). While astronomic time has a larger range,
we
can cross that bridge with a similar approach (e.g., an extension field
that
represents decades).

Patrick

-----Original Message-----
From: Michael T. Black [mailto:mtblack@berkeley.edu]
Sent: Wednesday, August 24, 2011 1:59 PM
To: Patrick Schmitz
Cc: Aron Roberts; talk@lists.collectionspace.org; Paolo Pellegatti
Subject: Re: [Talk] what's the best way to represent structured dates?

Hi Patrick,

Just concluded a summit of the chronometric working

group here at the Hearst (or, as we call it, me and Paolo),
and we're happy with a 300Myr limit (BTW-we and presumably
paleo, geology, etc. would call this BP, not BCE).  We have a
handful of 50-150Myr BP objects for which we might
conceivably have dates we'd want to enter.  However, we can't
think of any cases in which we've got objects older than
300Myr for which the = 300Myr date would have enough
importance to make entry of these dates a requirement for us
at the moment (e.g., we might have a bead made of an
Ordovician brachiopod, but the collection and production
dates would be the critical ones for us, not the date of the
material from which the item was made).  I say "at the
moment" because it would be awfully handy to have these dates
entered once we're at the point where we can offer federated
searching across collections/museums.

I can tell you, though, that you'll probably have a

tough time convincing the paleo folks that they don't need to
enter dates for their Carboniferous and earlier collections:

http://www.ucmp.berkeley.edu/science/inverts/carboniferous.php
http://www.ucmp.berkeley.edu/science/inverts/pennsylvanian.php
http://www.ucmp.berkeley.edu/science/inverts/mississippian.php
http://www.ucmp.berkeley.edu/science/inverts/devonian.php
http://www.ucmp.berkeley.edu/science/inverts/silurian.php
http://www.ucmp.berkeley.edu/science/inverts/ordovician.php
http://www.ucmp.berkeley.edu/science/inverts/cambrian.php
http://www.ucmp.berkeley.edu/science/inverts/precambrian.php

Michael

On Aug 24, 2011, at 12:00 PM, Patrick Schmitz wrote:

I think there is lots of potential for fuzzy date

interpretation, and

we can pursue this as a second step.

Next, we need to find out if the 300M yr BCE range will work for
Michael, for Paleo, Essig, et al. I am worried about Paleo

in particular.

Any other folks on this list with collections that go back

a long way?

How much of a limitation would the 300M yr BCE range be?

Thanks - Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf Of Aron
Roberts
Sent: Wednesday, August 24, 2011 11:39 AM
To: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured dates?

On Wed, Aug 24, 2011 at 11:16 AM, Janice Eklund
janice.l.eklund@gmail.com wrote:

I see no problem with the scalar date scenarios listed below.

Thanks much, Patrick, for such a simple, sensible way of creating
single range start and end values, and Jan for your feedback that
this would work for you.

I've had many long discussions with Susan and Glen about

this topic

and their frustrations with 'black box' or otherwise

unsatisfactory

fuzzy date algorithms.  Using our current model / schema for
structured dates, with its separate Year/Month/Day fields

for range

start/single date and end date - for which they both get

considerable

credit - and the clear logic you outline below, date range

search can

be transparent and easy to understand.

Note that entry of Year/Month/Day values versus

uncertainty in the

structure date popup is an either/or, as I understand it.  If you
want the transparent, easy-to-understand behavior, you

enter a year,

and maybe also a month, or month and day, for the

start/single date,

and
(optionally) for the end date.  If you enter uncertainty-related
values, when this is implemented down the road, perhaps a

pluggable

fuzzy date handler would be invoked, a la Dick's suggestion?

Aron

On 8/24/11, Patrick Schmitz pschmitz@berkeley.edu wrote:

Maybe I am missing something, but is there a problem with

the following:

Single date, expressed as "1910". We model this (in earliest and
latest scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910"

Date range expressed as earliest:"1910", latest:"1915". We

model this

as
esd: "Jan 1 1910" and lsd: "Dec 31 1915"

Date range expressed as earliest:"Jan 1910", latest:"Aug

1910". We

model this as esd: "Jan 1 1910" and lsd: "Aug 31 1910"

The esd and lsd scalar dates are used for search and

ordering, not display.

Patrick

-----Original Message-----
From: talk-bounces@lists.collectionspace.org
[mailto:talk-bounces@lists.collectionspace.org] On Behalf

Of Janice

Eklund
Sent: Wednesday, August 24, 2011 10:17 AM
To: Susan Stone
Cc: talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured dates?

I second Susan's concern about false precision.  At least

90% of our

dates are year only single dates or date ranges. If you are
considering fuzzy logic down the road, look at the

existing code

carefully.  The fuzzy date machine, at least as

implemented at UCB

for the HAVRC, does not handle ancient dates properly.

Jan

On 8/24/11, Susan Stone sstone@socrates.berkeley.edu wrote:

Patrick,

There was quite a bit of discussion of this in our

Wednesday afternoon

meetings at Berkeley that I'm sure Aron will recall. I

don't think

date fields go back far enough in time, so I'd recommend using
integers or floating point numbers and providing the math

in search.

It is also important to make sure that false precision isn't
introduced. If something is dated 1900, the system

shouldn't enter

that as January 1, 1900 unless the uncertainty values can

be used to

differentiate and the display date can be simply 1900.

Susan

On 08/24/2011 09:16 AM, Patrick Schmitz wrote:

Really? I am a little surprised that in all the design
discussions for structured dates this did not come up.
Anyway, I will say that we do not have the time to build

some system

that allows for a lot of custom behavior, or elaborate

heuristics

here (i.e., trying to divine what is intended from a

set of text

fields). I would strongly suggest that we either

  1. require entry of some actual date fields upon which

the application

 can reason,
 or
  1. we define a very simple set of rules for building such

date values

 from some of the existing text fields. By very simple,

I mean that

 we only consider fields like year, month, day, and

that the rules

 produce two scalar dates (early and late), which

either define a

 range, or define a point (if they are equal). AIUI, a

structured

 date /cannot/ model the idea of "some time/any

time after (or

 before) a date".

One further question is the semantic of which date (if

any) will be

used for ordering.
If and when advanced search supports date ranges (which

also implies

support for /before/ and /after/ logic in search), we can

handle the

interval logic associated with the structured dates, but

only if we

have a pair of scalar date values.
Down the road, I do think it would be interesting to

implement some

fuzzier logic on date search, to better support "circa"

logic and the

uncertainty values. However, I suspect that it will

take longer

to arrive at consensus on these semantics, and would

defer that

discussion for now.
Thanks - Patrick



 *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu]
 *Sent:* Wednesday, August 24, 2011 9:01 AM
 *To:* Patrick Schmitz
 *Cc:* 'Christopher Pott'; 'Aron Roberts';

talk@lists.collectionspace.org
Subject: Re: [Talk] what's the best way to represent

structured

dates?

 Hi Patrick, I think it's fair to say there's no

consensus because

 there's been no discussion :-) It's possible that each

customer will

 want to use structured dates differently, which will

make it hard to

 add any intelligence or even predictability to things

like search.

 However, I also think that if a significant number of

deploying
institutions can come to some consensus on best

ways to use

and
search structured dates, then many institutions will

choose to adopt

 the same best practices.

 Thanks,
 Chris

 On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote:
 Aron was out yesterday, and Rick is out for a few

days. They have

 done a lot of the structured dates work to date.
 The last I heard, Aron was hoping to get consensus

from the users

 about how we map all the text fields to actual date

fields that we

 can reason on (for search, ordering, etc). To date,

there seems to

 be no such consensus, which will preclude search,

ordering, etc.

 on structured dates. I would think that this would

make structured

 dates quite a bit less useful...
 Perhaps (I hope) my understanding is out of date, and

we just need

 to implement the logic. Perhaps Aron can jump in this

AM, when he

 gets in.
 Patrick


     *From:* talk-bounces@lists.collectionspace.org
     <mailto:talk-bounces@lists.collectionspace.org>
     [mailto:talk-bounces@lists.collectionspace.org]

*On Behalf Of

     *Chris Hoffman
     *Sent:* Tuesday, August 23, 2011 11:05 AM
     *To:* Christopher Pott
     *Cc:* talk@lists.collectionspace.org
     <mailto:talk@lists.collectionspace.org>
     *Subject:* Re: [Talk] what's the best way to

represent

     structured dates?

     Hi Chris,

     Thanks for asking these questions. We've been

wondering the

     same things. Have you changed any existing

calendar date

     fields to structured dates or added new

structured date

     fields? I think this is all quite new, and

together we might

     be able to come up with some best practices

(related to data

     migration and data entry).

     I suspect that search and advanced search of

structured date

     fields is going to be an issue for those of

us deploying

     CollectionSpace. Our users will want CSpace to be

smart about

     structured dates, interpreting qualifiers, and so

on, but I

     bet that we won't see much as much as we

would like in

the
first version of advanced search. I'm always

happy to be

     proved wrong! :-)

     Chris

     On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote:
     Hi,

     I'm looking at the structured date schema and

the best way to

     represent dates migrated from our old system.

Any information

     about the following would be much appreciated..

     1. I see some different/overlapping

possibilities for

     recording an object creation date. For example,

to represent

     "1718" (display date) one could either store:

     a) a single date, providing year only (1718)

     b) an earliest date (01.01.1718) with a latest date

(31.12.1718)

     c) a single date (01.01.1718) with a

qualifier of + 364

days

     Is there any difference in the meaning of these

three (and

     would they all be treated equally by

services such as

     advanced search)? Our migration data is

currently in format

     'b' but the other formats are sometimes

'friendlier' for new

     input.

     2. How are date "certainties" and

"qualifiers" used by

     advanced search?

     I've specified "1718", as a creation date with a

certainty of

     "circa". Will a search for objects created

during 1717 result

     in a hit?

     Or is it necessary to use qualifiers (eg. 1718

+/- 2 years)

     to define exact limits for this type of searching?

     3. Is the Date schema locally extendable? (We

need a field to

     hold an English translation of the display text)

     4. Will Advanced search by creation date

also search

the
display date or will it just use the

single/earliest

and
latest date groups?

     Thanks,

     Chris

     _______________________________________________
     Talk mailing list
     Talk@lists.collectionspace.org
     <mailto:Talk@lists.collectionspace.org>

tionspace.org


Talk mailing list
Talk@lists.collectionspace.org

i

onspace.org


Talk mailing list
Talk@lists.collectionspace.org

Thanks for the responses. Can you tell me if these interpretations of your responses are correct.... * if we always provide a full (dd-MM-yyyy GG) earliest and latest date, these will be the dates used by services for search and ordering * if we provide less information (such as just a year in earliest date) services will generate an internal earliest and latest date to cover that period * Services won't be parsing display dates to generate earliest and latest dates * Services won't be examining 'qualifiers' or 'certainty' fields so if we rely on these then searching will fail Is the date schema extensible? We need a way to store some information with a date, which probably wouldn't be useful to services but that can help us modify the UI to our requirements. Our earliest structured date is currently 1600 BC which I thought was quite old until I read the other posts. Just out of interest to other implementers, some screen shots of our planned date UI implementation can be seen here http://wiki.collectionspace.org/display/collectionspace/Date+and+Time+Use+Cases. It will use a combination of drop down lists and day/month/year inputs to enable generation of display dates according to some precise rules (required for catalogues and signs). It should generate a translation at the same time, as well as the earliest and latest dates. Cheers, Chris > -----Oprindelig meddelelse----- > Fra: talk-bounces@lists.collectionspace.org [mailto:talk- > bounces@lists.collectionspace.org] På vegne af Patrick Schmitz > Sendt: 24. august 2011 23:47 > Til: 'Michael T. Black' > Cc: talk@lists.collectionspace.org > Emne: Re: [Talk] what's the best way to represent structured dates? > > So this is what I more or less expected. > > I see several alternatives: > > 1) We do something special for Paleo > 2) We implement our own Calendar support that can get more range out of a > Long storage unit. > > 2 would be a lot more work, and I am not thrilled with that approach. > > I am pretty sure I do not know enough about how Paleo and friends enter > dates, interpret dates, compare them, etc., etc. If they actually use a > different model for these dates, then perhaps 1 is a reasonable approach. > E.g., we could support a different scalar date field that used a different > unit (like years, rather than days), and then they could do searches on > that > rather than date fields. A long range of years into the past (E.g., rooted > at 3000 CE) could represent years nearly back to the beginning of our > planet > (about 4.2 billion years ago). While astronomic time has a larger range, > we > can cross that bridge with a similar approach (e.g., an extension field > that > represents decades). > > Patrick > > > -----Original Message----- > > From: Michael T. Black [mailto:mtblack@berkeley.edu] > > Sent: Wednesday, August 24, 2011 1:59 PM > > To: Patrick Schmitz > > Cc: Aron Roberts; talk@lists.collectionspace.org; Paolo Pellegatti > > Subject: Re: [Talk] what's the best way to represent structured dates? > > > > Hi Patrick, > > > > Just concluded a summit of the chronometric working > > group here at the Hearst (or, as we call it, me and Paolo), > > and we're happy with a 300Myr limit (BTW-we and presumably > > paleo, geology, etc. would call this BP, not BCE). We have a > > handful of 50-150Myr BP objects for which we might > > conceivably have dates we'd want to enter. However, we can't > > think of any cases in which we've got objects older than > > 300Myr for which the = 300Myr date would have enough > > importance to make entry of these dates a requirement for us > > at the moment (e.g., we might have a bead made of an > > Ordovician brachiopod, but the collection and production > > dates would be the critical ones for us, not the date of the > > material from which the item was made). I say "at the > > moment" because it would be awfully handy to have these dates > > entered once we're at the point where we can offer federated > > searching across collections/museums. > > > > I can tell you, though, that you'll probably have a > > tough time convincing the paleo folks that they don't need to > > enter dates for their Carboniferous and earlier collections: > > > > http://www.ucmp.berkeley.edu/science/inverts/carboniferous.php > > http://www.ucmp.berkeley.edu/science/inverts/pennsylvanian.php > > http://www.ucmp.berkeley.edu/science/inverts/mississippian.php > > http://www.ucmp.berkeley.edu/science/inverts/devonian.php > > http://www.ucmp.berkeley.edu/science/inverts/silurian.php > > http://www.ucmp.berkeley.edu/science/inverts/ordovician.php > > http://www.ucmp.berkeley.edu/science/inverts/cambrian.php > > http://www.ucmp.berkeley.edu/science/inverts/precambrian.php > > > > Michael > > > > On Aug 24, 2011, at 12:00 PM, Patrick Schmitz wrote: > > > > > I think there is lots of potential for fuzzy date > > interpretation, and > > > we can pursue this as a second step. > > > > > > Next, we need to find out if the 300M yr BCE range will work for > > > Michael, for Paleo, Essig, et al. I am worried about Paleo > > in particular. > > > > > > Any other folks on this list with collections that go back > > a long way? > > > How much of a limitation would the 300M yr BCE range be? > > > > > > Thanks - Patrick > > > > > >> -----Original Message----- > > >> From: talk-bounces@lists.collectionspace.org > > >> [mailto:talk-bounces@lists.collectionspace.org] On Behalf Of Aron > > >> Roberts > > >> Sent: Wednesday, August 24, 2011 11:39 AM > > >> To: talk@lists.collectionspace.org > > >> Subject: Re: [Talk] what's the best way to represent > > structured dates? > > >> > > >> On Wed, Aug 24, 2011 at 11:16 AM, Janice Eklund > > >> <janice.l.eklund@gmail.com> wrote: > > >>> I see no problem with the scalar date scenarios listed below. > > >> > > >> Thanks much, Patrick, for such a simple, sensible way of creating > > >> single range start and end values, and Jan for your feedback that > > >> this would work for you. > > >> > > >> I've had many long discussions with Susan and Glen about > > this topic > > >> and their frustrations with 'black box' or otherwise > > unsatisfactory > > >> fuzzy date algorithms. Using our current model / schema for > > >> structured dates, with its separate Year/Month/Day fields > > for range > > >> start/single date and end date - for which they both get > > considerable > > >> credit - and the clear logic you outline below, date range > > search can > > >> be transparent and easy to understand. > > >> > > >> Note that entry of Year/Month/Day values versus > > uncertainty in the > > >> structure date popup is an either/or, as I understand it. If you > > >> want the transparent, easy-to-understand behavior, you > > enter a year, > > >> and maybe also a month, or month and day, for the > > start/single date, > > >> and > > >> (optionally) for the end date. If you enter uncertainty-related > > >> values, when this is implemented down the road, perhaps a > > pluggable > > >> fuzzy date handler would be invoked, a la Dick's suggestion? > > >> > > >> Aron > > >> > > >>> On 8/24/11, Patrick Schmitz <pschmitz@berkeley.edu> wrote: > > >>>> Maybe I am missing something, but is there a problem with > > >> the following: > > >>>> > > >>>> Single date, expressed as "1910". We model this (in earliest and > > >>>> latest scalar dates) as esd: "Jan 1 1910" and lsd: "Dec 31 1910" > > >>>> > > >>>> Date range expressed as earliest:"1910", latest:"1915". We > > >> model this > > >>>> as > > >>>> esd: "Jan 1 1910" and lsd: "Dec 31 1915" > > >>>> > > >>>> Date range expressed as earliest:"Jan 1910", latest:"Aug > > 1910". We > > >>>> model this as esd: "Jan 1 1910" and lsd: "Aug 31 1910" > > >>>> > > >>>> The esd and lsd scalar dates are used for search and > > >> ordering, not display. > > >>>> > > >>>> Patrick > > >>>> > > >>>>> -----Original Message----- > > >>>>> From: talk-bounces@lists.collectionspace.org > > >>>>> [mailto:talk-bounces@lists.collectionspace.org] On Behalf > > >> Of Janice > > >>>>> Eklund > > >>>>> Sent: Wednesday, August 24, 2011 10:17 AM > > >>>>> To: Susan Stone > > >>>>> Cc: talk@lists.collectionspace.org > > >>>>> Subject: Re: [Talk] what's the best way to represent > > >> structured dates? > > >>>>> > > >>>>> I second Susan's concern about false precision. At least > > >> 90% of our > > >>>>> dates are year only single dates or date ranges. If you are > > >>>>> considering fuzzy logic down the road, look at the > > existing code > > >>>>> carefully. The fuzzy date machine, at least as > > >> implemented at UCB > > >>>>> for the HAVRC, does not handle ancient dates properly. > > >>>>> > > >>>>> Jan > > >>>>> > > >>>>> On 8/24/11, Susan Stone <sstone@socrates.berkeley.edu> wrote: > > >>>>>> Patrick, > > >>>>>> > > >>>>>> There was quite a bit of discussion of this in our > > >>>>> Wednesday afternoon > > >>>>>> meetings at Berkeley that I'm sure Aron will recall. I > > >> don't think > > >>>>>> date fields go back far enough in time, so I'd recommend using > > >>>>>> integers or floating point numbers and providing the math > > >>>>> in search. > > >>>>>> It is also important to make sure that false precision isn't > > >>>>>> introduced. If something is dated 1900, the system > > >> shouldn't enter > > >>>>>> that as January 1, 1900 unless the uncertainty values can > > >>>>> be used to > > >>>>>> differentiate and the display date can be simply 1900. > > >>>>>> > > >>>>>> Susan > > >>>>>> > > >>>>>> > > >>>>>> On 08/24/2011 09:16 AM, Patrick Schmitz wrote: > > >>>>>>> Really? I am a little surprised that in all the design > > >>>>>>> discussions for structured dates this did not come up. > > >>>>>>> Anyway, I will say that we do not have the time to build > > >>>>> some system > > >>>>>>> that allows for a lot of custom behavior, or elaborate > > >> heuristics > > >>>>>>> here (i.e., trying to divine what is intended from a > > >> set of text > > >>>>>>> fields). I would strongly suggest that we either > > >>>>>>> > > >>>>>>> 1. > > >>>>>>> require entry of some actual date fields upon which > > >>>>> the application > > >>>>>>> can reason, > > >>>>>>> or > > >>>>>>> 2. > > >>>>>>> we define a very simple set of rules for building such > > >>>>> date values > > >>>>>>> from some of the existing text fields. By very simple, > > >>>>> I mean that > > >>>>>>> we only consider fields like year, month, day, and > > >>>>> that the rules > > >>>>>>> produce two scalar dates (early and late), which > > >>>>> either define a > > >>>>>>> range, or define a point (if they are equal). AIUI, a > > >>>>> structured > > >>>>>>> date /cannot/ model the idea of "some time/any > > >> time after (or > > >>>>>>> before) a date". > > >>>>>>> > > >>>>>>> One further question is the semantic of which date (if > > >>>>> any) will be > > >>>>>>> used for ordering. > > >>>>>>> If and when advanced search supports date ranges (which > > >>>>> also implies > > >>>>>>> support for /before/ and /after/ logic in search), we can > > >>>>> handle the > > >>>>>>> interval logic associated with the structured dates, but > > >>>>> only if we > > >>>>>>> have a pair of scalar date values. > > >>>>>>> Down the road, I do think it would be interesting to > > >>>>> implement some > > >>>>>>> fuzzier logic on date search, to better support "circa" > > >>>>> logic and the > > >>>>>>> uncertainty values. However, I suspect that it will > > >> take longer > > >>>>>>> to arrive at consensus on these semantics, and would > > >> defer that > > >>>>>>> discussion for now. > > >>>>>>> Thanks - Patrick > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>> -------------------------------------------------------------- > > >>>>> ---------- > > >>>>>>> *From:* Chris Hoffman [mailto:chris.hoffman@berkeley.edu] > > >>>>>>> *Sent:* Wednesday, August 24, 2011 9:01 AM > > >>>>>>> *To:* Patrick Schmitz > > >>>>>>> *Cc:* 'Christopher Pott'; 'Aron Roberts'; > > >>>>>>> talk@lists.collectionspace.org > > >>>>>>> *Subject:* Re: [Talk] what's the best way to represent > > >>>>> structured > > >>>>>>> dates? > > >>>>>>> > > >>>>>>> Hi Patrick, I think it's fair to say there's no > > >>>>> consensus because > > >>>>>>> there's been no discussion :-) It's possible that each > > >>>>> customer will > > >>>>>>> want to use structured dates differently, which will > > >>>>> make it hard to > > >>>>>>> add any intelligence or even predictability to things > > >>>>> like search. > > >>>>>>> However, I also think that if a significant number of > > >>>>>>> deploying > > >>>>>>> institutions can come to some consensus on best > > >> ways to use > > >>>>>>> and > > >>>>>>> search structured dates, then many institutions will > > >>>>> choose to adopt > > >>>>>>> the same best practices. > > >>>>>>> > > >>>>>>> Thanks, > > >>>>>>> Chris > > >>>>>>> > > >>>>>>> On Aug 24, 2011, at 8:55 AM, Patrick Schmitz wrote: > > >>>>>>> > > >>>>>>>> Aron was out yesterday, and Rick is out for a few > > >>>>> days. They have > > >>>>>>>> done a lot of the structured dates work to date. > > >>>>>>>> The last I heard, Aron was hoping to get consensus > > >>>>> from the users > > >>>>>>>> about how we map all the text fields to actual date > > >>>>> fields that we > > >>>>>>>> can reason on (for search, ordering, etc). To date, > > >>>>> there seems to > > >>>>>>>> be no such consensus, which will preclude search, > > >>>>> ordering, etc. > > >>>>>>>> on structured dates. I would think that this would > > >>>>> make structured > > >>>>>>>> dates quite a bit less useful... > > >>>>>>>> Perhaps (I hope) my understanding is out of date, and > > >>>>> we just need > > >>>>>>>> to implement the logic. Perhaps Aron can jump in this > > >>>>> AM, when he > > >>>>>>>> gets in. > > >>>>>>>> Patrick > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>> -------------------------------------------------------------- > > >>>>> ---------- > > >>>>>>>> *From:* talk-bounces@lists.collectionspace.org > > >>>>>>>> <mailto:talk-bounces@lists.collectionspace.org> > > >>>>>>>> [mailto:talk-bounces@lists.collectionspace.org] > > >>>>> *On Behalf Of > > >>>>>>>> *Chris Hoffman > > >>>>>>>> *Sent:* Tuesday, August 23, 2011 11:05 AM > > >>>>>>>> *To:* Christopher Pott > > >>>>>>>> *Cc:* talk@lists.collectionspace.org > > >>>>>>>> <mailto:talk@lists.collectionspace.org> > > >>>>>>>> *Subject:* Re: [Talk] what's the best way to > > represent > > >>>>>>>> structured dates? > > >>>>>>>> > > >>>>>>>> Hi Chris, > > >>>>>>>> > > >>>>>>>> Thanks for asking these questions. We've been > > >>>>> wondering the > > >>>>>>>> same things. Have you changed any existing > > >> calendar date > > >>>>>>>> fields to structured dates or added new > > >> structured date > > >>>>>>>> fields? I think this is all quite new, and > > >>>>> together we might > > >>>>>>>> be able to come up with some best practices > > >>>>> (related to data > > >>>>>>>> migration and data entry). > > >>>>>>>> > > >>>>>>>> I suspect that search and advanced search of > > >>>>> structured date > > >>>>>>>> fields is going to be an issue for those of > > >> us deploying > > >>>>>>>> CollectionSpace. Our users will want CSpace to be > > >>>>> smart about > > >>>>>>>> structured dates, interpreting qualifiers, and so > > >>>>> on, but I > > >>>>>>>> bet that we won't see much as much as we > > >> would like in > > >>>>>>>> the > > >>>>>>>> first version of advanced search. I'm always > > >> happy to be > > >>>>>>>> proved wrong! :-) > > >>>>>>>> > > >>>>>>>> Chris > > >>>>>>>> > > >>>>>>>> On Aug 23, 2011, at 8:48 AM, Christopher Pott wrote: > > >>>>>>>> > > >>>>>>>>> Hi, > > >>>>>>>>> > > >>>>>>>>> I'm looking at the structured date schema and > > >>>>> the best way to > > >>>>>>>>> represent dates migrated from our old system. > > >>>>> Any information > > >>>>>>>>> about the following would be much appreciated.. > > >>>>>>>>> > > >>>>>>>>> 1. I see some different/overlapping > > possibilities for > > >>>>>>>>> recording an object creation date. For example, > > >>>>> to represent > > >>>>>>>>> "1718" (display date) one could either store: > > >>>>>>>>> > > >>>>>>>>> a) a single date, providing year only (1718) > > >>>>>>>>> > > >>>>>>>>> b) an earliest date (01.01.1718) with a latest date > > >>>>>>>>> (31.12.1718) > > >>>>>>>>> > > >>>>>>>>> c) a single date (01.01.1718) with a > > >> qualifier of + 364 > > >>>>>>>>> days > > >>>>>>>>> > > >>>>>>>>> Is there any difference in the meaning of these > > >>>>> three (and > > >>>>>>>>> would they all be treated equally by > > services such as > > >>>>>>>>> advanced search)? Our migration data is > > >>>>> currently in format > > >>>>>>>>> 'b' but the other formats are sometimes > > >>>>> 'friendlier' for new > > >>>>>>>>> input. > > >>>>>>>>> > > >>>>>>>>> 2. How are date "certainties" and > > >> "qualifiers" used by > > >>>>>>>>> advanced search? > > >>>>>>>>> > > >>>>>>>>> I've specified "1718", as a creation date with a > > >>>>> certainty of > > >>>>>>>>> "circa". Will a search for objects created > > >>>>> during 1717 result > > >>>>>>>>> in a hit? > > >>>>>>>>> > > >>>>>>>>> Or is it necessary to use qualifiers (eg. 1718 > > >>>>> +/- 2 years) > > >>>>>>>>> to define exact limits for this type of searching? > > >>>>>>>>> > > >>>>>>>>> 3. Is the Date schema locally extendable? (We > > >>>>> need a field to > > >>>>>>>>> hold an English translation of the display text) > > >>>>>>>>> > > >>>>>>>>> 4. Will Advanced search by creation date > > also search > > >>>>>>>>> the > > >>>>>>>>> display date or will it just use the > > single/earliest > > >>>>>>>>> and > > >>>>>>>>> latest date groups? > > >>>>>>>>> > > >>>>>>>>> Thanks, > > >>>>>>>>> > > >>>>>>>>> Chris > > >>>>>>>>> > > >>>>>>>>> _______________________________________________ > > >>>>>>>>> Talk mailing list > > >>>>>>>>> Talk@lists.collectionspace.org > > >>>>>>>>> <mailto:Talk@lists.collectionspace.org> > > >>>>>>>>> > > >>>>>>>>> > > >>>>> > > >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.collec > > >>>>>>>>> tionspace.org > > >>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> _______________________________________________ > > >>>>>>> Talk mailing list > > >>>>>>> Talk@lists.collectionspace.org > > >>>>>>> > > >>>>> > > >> > > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collect > > >>>>> i > > >>>>>>> onspace.org > > >>>>>> > > >>>>>> > > >>>>>> _______________________________________________ > > >>>>>> Talk mailing list > > >>>>>> Talk@lists.collectionspace.org > > >>>>>> > > >>>>> > > >> > > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collect > > >>>>> io > > >>>>>> nspace.org > > >>>>>> > > >>>>> > > >>>>> _______________________________________________ > > >>>>> Talk mailing list > > >>>>> Talk@lists.collectionspace.org > > >>>>> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c > > >>>>> ollectionspace.org > > >>>>> > > >>>> > > >>>> > > >>> > > >>> _______________________________________________ > > >>> Talk mailing list > > >>> Talk@lists.collectionspace.org > > >>> > > >> > > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collecti > > >> o > > >>> nspace.org > > >>> > > >> > > >> _______________________________________________ > > >> Talk mailing list > > >> Talk@lists.collectionspace.org > > >> http://lists.collectionspace.org/mailman/listinfo/talk_lists.c > > >> ollectionspace.org > > >> > > > > > > > > > > _______________________________________________ > Talk mailing list > Talk@lists.collectionspace.org > http://lists.collectionspace.org/mailman/listinfo/talk_lists.collectionspa > ce.org