Dataset Integration Hack - Revision history

JamieNorrish: /* Platform */ Added some details about OAI-PMH

2010-11-16T19:05:09Z

Platform: Added some details about OAI-PMH

← Older revision		Revision as of 20:05, 16 November 2010
Line 9:		Line 9:
	OAI-PMH server and DC metadata. (JN, MR, JMV: more info please?)		OAI-PMH server and DC metadata. (JN, MR, JMV: more info please?)

	[http://www.dlese.org/dds/services/joai_software.jsp JOAI] is a Java implementation of OAI-PMH data provider and harvester that ~~might be used~~ for ~~a first proof-of-concept implementation~~.		An OAI-PMH server is the natural solution to having metadata records that need to be served to clients. It allows for clients to search on those records in a variety of ways (importantly including 'records modified since date') and to perform automatic translation of records from one format to another.

			If there are metadata records out in the wild, the harvester can fetch those and merge them into its set of records. This is particularly useful if any of the datasets change over time, since only the modified records need to be fetched and updated. If those records aren't being exposed via OAI-PMH, or don't change, then a custom harvester/converter is needed.

			Dublin Core metadata is not the only metadata format that an OAI-PMH server can serve. It is worth spending some time to determine what information is required and use or create an appropriate schema; if it isn't Dublin Core, then a fallback DC can be provided as well.

			[http://www.dlese.org/dds/services/joai_software.jsp JOAI] is a Java implementation of OAI-PMH data provider and harvester that is both easy to set up and play around with, and is also suitable for production use.

	== Metadata ==		== Metadata ==

MatteoRomanello: /* What's next? */

2010-11-16T15:02:57Z

What's next?

← Older revision		Revision as of 16:02, 16 November 2010
Line 81:		Line 81:
	* Create sample metadata for each dataset (ideally by writing scripts for the sake of process reproducibility)		* Create sample metadata for each dataset (ideally by writing scripts for the sake of process reproducibility)
	* discuss viability of [http://www.ckan.net/ CKAN] for our purposes		* discuss viability of [http://www.ckan.net/ CKAN] for our purposes
			* provide a description of how we generate the metdata we agreed on for each dataset
	* Next meeting will be on 17/11/2010 1pm-2pm (CCH, seminar room).		* Next meeting will be on 17/11/2010 1pm-2pm (CCH, seminar room).

MatteoRomanello: /* What's next? */

2010-11-16T15:01:50Z

What's next?

← Older revision		Revision as of 16:01, 16 November 2010
Line 81:		Line 81:
	* Create sample metadata for each dataset (ideally by writing scripts for the sake of process reproducibility)		* Create sample metadata for each dataset (ideally by writing scripts for the sake of process reproducibility)
	* discuss viability of [http://www.ckan.net/ CKAN] for our purposes		* discuss viability of [http://www.ckan.net/ CKAN] for our purposes
	* Next meeting will be on 17/11/2010 1pm-2pm.		* Next meeting will be on 17/11/2010 1pm-2pm (CCH, seminar room).

MatteoRomanello: /* What's next? */

2010-11-16T15:01:04Z

What's next?

← Older revision		Revision as of 16:01, 16 November 2010
Line 81:		Line 81:
	* Create sample metadata for each dataset (ideally by writing scripts for the sake of process reproducibility)		* Create sample metadata for each dataset (ideally by writing scripts for the sake of process reproducibility)
	* discuss viability of [http://www.ckan.net/ CKAN] for our purposes		* discuss viability of [http://www.ckan.net/ CKAN] for our purposes
	* Next meeting.		* Next meeting will be on 17/11/2010 1pm-2pm.

MatteoRomanello: /* What's next? */

2010-11-11T17:21:18Z

What's next?

← Older revision		Revision as of 18:21, 11 November 2010
Line 80:		Line 80:
	* Set up OAIPMH server.		* Set up OAIPMH server.
	* Create sample metadata for each dataset (ideally by writing scripts for the sake of process reproducibility)		* Create sample metadata for each dataset (ideally by writing scripts for the sake of process reproducibility)
	* discuss viability of CKAN for our purposes		* discuss viability of [http://www.ckan.net/ CKAN] for our purposes
	* Next meeting.		* Next meeting.

MatteoRomanello: /* Platform */

2010-11-11T17:20:21Z

Platform

← Older revision		Revision as of 18:20, 11 November 2010
Line 8:		Line 8:

	OAI-PMH server and DC metadata. (JN, MR, JMV: more info please?)		OAI-PMH server and DC metadata. (JN, MR, JMV: more info please?)

			[http://www.dlese.org/dds/services/joai_software.jsp JOAI] is a Java implementation of OAI-PMH data provider and harvester that might be used for a first proof-of-concept implementation.

	== Metadata ==		== Metadata ==

MatteoRomanello: /* What's next? */

2010-11-11T17:16:38Z

What's next?

← Older revision		Revision as of 18:16, 11 November 2010
Line 77:		Line 77:

	* Set up OAIPMH server.		* Set up OAIPMH server.
	* Create sample metadata for each dataset.		* Create sample metadata for each dataset (ideally by writing scripts for the sake of process reproducibility)
			* discuss viability of CKAN for our purposes
	* Next meeting.		* Next meeting.

MatteoRomanello: /* What's next? */

2010-11-11T17:14:46Z

What's next?

← Older revision		Revision as of 18:14, 11 November 2010
Line 76:		Line 76:
	== What's next? ==		== What's next? ==

	Set up OAIPMH server.		* Set up OAIPMH server.
			* Create sample metadata for each dataset.
	Create sample metadata for each dataset.		* Next meeting.

	Next meeting.

MatteoRomanello: /* Metadata */

2010-11-11T17:14:03Z

Metadata

← Older revision		Revision as of 18:14, 11 November 2010
Line 10:		Line 10:

	== Metadata ==		== Metadata ==

			=== Extraction ===

			Metadata will be extracted on a case-by-case basis from the source data, with additional global parameters provided from local knowledge as required. Ideally, and eventually, individual datasets would provide their own OAI service to expose this metadata. (We may try to illustrate this with IAph and IRT at some point.)

	=== Harvesting ===		=== Harvesting ===

	~~Metadata~~ will be ~~harvested on~~ a ~~case-~~by~~-case basis from~~ the ~~source data,~~ with ~~additional global parameters provided from local knowledge as required. Ideally, and eventually, individual datasets would provide their own~~ OAI ~~service to expose this metadata. (We may try to illustrate this with IAph and IRT at some point~~.)		Each dataset will be essentially transformed into a data provider by exposing the extracted metadata accordingly with the OAI-PMH.

	=== Schema ===		=== Schema ===

GabrielBodard: /* Schema */ tabulated

2010-11-04T12:16:32Z

Schema: tabulated

← Older revision		Revision as of 13:16, 4 November 2010
Line 19:		Line 19:
	[http://www.openarchives.org/OAI/openarchivesprotocol.html#dublincore OAI-PMH in Dublin Core]		[http://www.openarchives.org/OAI/openarchivesprotocol.html#dublincore OAI-PMH in Dublin Core]

	* dc:title		{\| border="1"
	** title of resource		\|+
	* dc:creator		\|'''Tags'''
	** harvest (or known?)		\|'''How we generate?'''
	* dc:subject		\|-
	** ??		\|dc:title
	* dc:description		\|title of resource
	** if any free prose		\|-
	* dc:publisher		\|dc:creator
	** harvest		\|harvest (or known?)
	* dc:contributor		\|-
	** harvest if given		\|dc:subject
	* dc:date		\|??
	** harvest		\|-
	* dc:type		\|dc:description
	** closed list (edition\|photograph\|commentary\|database\|linked data\|other)		\|if any free prose
	* dc:format		\|-
	** filetypes?		\|dc:publisher
	* dc:identifier		\|harvest
	** URI and/or URL?		\|-
	* dc:source		\|dc:contributor
	** ??		\| harvest if given
	* dc:language		\|-
	** = modern language		\|dc:date
	* dc:relation		\| harvest
	** ??		\|-
	* dc:coverage		\|dc:type
	** ??		\| closed list (edition\|photograph\|commentary\|database\|linked data\|other)
	* dc:rights		\|-
	** = license (in spreadsheet)		\|dc:format
			\| filetypes?
			\|-
			\|dc:identifier
			\| URI and/or URL?
			\|-
			\|dc:source
			\| ??
			\|-
			\|dc:language
			\| = modern language
			\|-
			\|dc:relation
			\| ??
			\|-
			\|dc:coverage
			\| ??
			\|-
			\|dc:rights
			\| = license (in spreadsheet)
			\|}

	== What's next? ==		== What's next? ==