# llms.txt for Oxford University Research Archive (ORA) # Base URL: https://ora.ox.ac.uk # Contact: ora-dev@bodleian.ox.ac.uk # Version: 1.0 # Last Updated: 2026-02-26 ---------------------------------------------------------------------- PURPOSE ---------------------------------------------------------------------- The Oxford University Research Archive (ORA) is the institutional repository of the University of Oxford. AI agents and automated harvesters are permitted to retrieve structured metadata and associated binary files strictly according to the guidance below. Structured machine interfaces are authoritative for harvesting. HTML pages are primarily for human access. ---------------------------------------------------------------------- RECORD DISCOVERY (MANDATORY) ---------------------------------------------------------------------- Agents MUST use sitemap-based discovery. 1. Retrieve: https://ora.ox.ac.uk/robots.txt 2. Extract sitemap URLs listed at the bottom of the file. 3. Parse XML sitemaps. 4. Extract metadata landing page URLs. 5. Extract UUID from landing page URL. Agents MUST NOT: - Use site search - Traverse browse interfaces - Generate deep query URLs - Perform recursive crawl discovery outside sitemap listings Full daily re-harvest of all records is unnecessary and discouraged. ---------------------------------------------------------------------- INCREMENTAL HARVEST (REQUIRED) ---------------------------------------------------------------------- Agents SHOULD use incremental harvesting via OAI-PMH: /ora_oaipmh_service/rioxx_terms_v3?verb=ListIdentifiers Agents MUST: - Use from/until parameters where possible - Use UTC timestamps - Harvest no more than once per 24 hours unless justified - Avoid re-harvesting unchanged UUIDs ---------------------------------------------------------------------- UUID FORMAT ---------------------------------------------------------------------- UUIDs are: - Prefixed: "uuid:" - Hyphenated RFC 4122 format - Case-sensitive - Treated as opaque strings Example: uuid:43af7268-d633-4bc7-8a0b-044acdb47131 Agents MUST NOT: - Remove prefix - Change case - Normalize formatting ---------------------------------------------------------------------- AUTHORITATIVE METADATA – OAI-PMH (MANDATORY) ---------------------------------------------------------------------- The RIOXX v3 OAI-PMH endpoint is the primary and authoritative source of structured metadata. For each UUID: /ora_oaipmh_service/rioxx_terms_v3?identifier=oai:ora.ox.ac.uk:uuid:{{UUID}}&verb=GetRecord RIOXX v3 provides the vast majority of useful metadata, including: - Descriptive metadata - File identifiers (download URIs) - Licensing information - Rights statements - Persistent identifiers Agents MUST treat OAI-PMH output as authoritative over landing page content in the event of discrepancy. ---------------------------------------------------------------------- SUPPLEMENTAL HTML METADATA (PERMITTED, LIMITED USE) ---------------------------------------------------------------------- Some metadata elements (including certain relationships and identifiers) may appear only in the human-readable metadata landing page. Agents MAY access the metadata landing page to retrieve such supplemental information when required. However: - OAI-PMH endpoints MUST remain the primary harvest mechanism. - HTML parsing MUST be limited and targeted. - HTML pages MUST NOT be used for large-scale structured harvesting when equivalent information is available via OAI-PMH. - Agents MUST NOT scrape layout-dependent or presentation-specific elements. - Agents MUST NOT increase crawl depth beyond the landing page. HTML access should be minimal, proportional, and necessary. ---------------------------------------------------------------------- DATASET RECORDS – DATACITE METADATA (MANDATORY) ---------------------------------------------------------------------- If the OAI-PMH RIOXX response contains a element whose value (case-insensitive) is: dataset then the record MUST be treated as a dataset record. Agents MUST retrieve additional metadata from: /ora_oaipmh_service/datacite_dc?identifier=oai:ora.ox.ac.uk:uuid:{{UUID}}&verb=GetRecord The DataCite 4.6 metadata output includes file size information for all associated binary files. Agents MUST inspect file size metadata before downloading any binary file. ---------------------------------------------------------------------- FILE SIZE & LARGE FILE POLICY (MANDATORY) ---------------------------------------------------------------------- If any file exceeds 10 MB: - No more than ONE file larger than 10 MB may be downloaded at a time. - Total concurrent file downloads MUST NOT exceed 10. - Parallel range requests for a single file are PROHIBITED. HEAD requests are permitted. Files exceeding 2 GB require prior contact with ORA before automated harvesting. ---------------------------------------------------------------------- BINARY FILE LOCATIONS ---------------------------------------------------------------------- Canonical binary file URLs for harvesters follow the format: /objects/UUID/files/FILEID Example: https://ora.ox.ac.uk/objects/uuid:cd4d605f-9f28-4ad8-b606-ea55588f8b1e/files/rqa9k41zd78b Other historical file URLs have been used in ORA and may appear in older resources. These URLs typically include the string "download". These have all been replaced by canonical binary file URLs. Agents MUST NOT use legacy or non-canonical URLs. Legacy URLs will include the UUID of the metadata record, which can be used to retrieve the canonical file URL. ---------------------------------------------------------------------- BINARY FILES: CONTENT STABILITY GUARANTEE ---------------------------------------------------------------------- - Binary files published at canonical URLs are immutable. - While publicly accessible at a canonical URL, a file’s byte content will not change. - New versions of files are published at new canonical URLs. - Canonical binary URLs may be treated as permanently content-stable resources. ---------------------------------------------------------------------- RESTRICTED / REQUEST-ONLY FILES (MANDATORY) ---------------------------------------------------------------------- Some binary files are marked as: - Request-only - Login-required - Restricted access Agents MUST NOT: - Attempt to bypass authentication - Attempt to automate “request access” workflows - Attempt to simulate user login - Attempt repeated access to restricted file URLs If access is required for legitimate research purposes, the controlling organisation of the harvesting agent MUST contact ora-dev@bodleian.ox.ac.uk. Such requests MUST be made by a human representative. Automated access to restricted materials is prohibited. ---------------------------------------------------------------------- CONCURRENCY LIMITS (MANDATORY) ---------------------------------------------------------------------- Metadata requests: - Maximum 5 concurrent connections - Maximum 1 request per second per IP Binary file downloads: - Maximum 10 concurrent downloads total - Maximum 1 concurrent file larger than 10 MB ---------------------------------------------------------------------- LOAD AWARENESS & BACKOFF (MANDATORY) ---------------------------------------------------------------------- Agents MUST monitor response times, especially for OAI-PMH metadata requests, for signs that the site is under load. Indicators may include: - Increased latency - 429 responses - 503 responses - 5xx errors - Connection resets Agents MUST implement exponential backoff when such indicators occur. Agents SHOULD dynamically reduce concurrency if response times increase. Aggressive retry loops are prohibited. ---------------------------------------------------------------------- LICENSING & RIGHTS ---------------------------------------------------------------------- Agents MUST: - Inspect file-level licence information in metadata - Respect rights statements - Not assume open licence unless explicitly stated - Attribute appropriately where required ---------------------------------------------------------------------- ENFORCEMENT ---------------------------------------------------------------------- Automated throttling or blocking may occur if server performance is affected. Persistent or severe violations may result in IP blocking. Reinstatement requires contacting: ora-dev@bodleian.ox.ac.uk ---------------------------------------------------------------------- CONTACT ---------------------------------------------------------------------- For large-scale harvesting, bulk dataset access, or feature requests: ora-dev@bodleian.ox.ac.uk