Start reworking handling of transcripts

.gitignore

@@ -35,3 +35,5 @@ static/data/bibItems/
 !/content/en/history/bibliography/_index.md
 !/content/en/history/bibliography/intake/
 !/content/en/history/bibliography/intake/**
+!/content/en/history/bibliography/zotero-resources/
+!/content/en/history/bibliography/zotero-resources/**

content/en/history/bibliography/zotero-resources/_index.md

@@ -0,0 +1,252 @@
+---
+title: "Zotero Type Mapping Reference"
+description: "How Interlisp Zotero bibliography types map to CSL JSON and Schema.org vocabulary"
+type: docs
+toc_hide: false
+---
+
+## Purpose
+
+This page documents how bibliography entry types are classified and mapped through the processing pipeline. It serves as a reference for:
+
+- **Metadata creators** contributing to the Interlisp Zotero group who want to understand how their entries will be categorized
+- **Developers** working on the bibliography processing system who need to know current type support
+- **Data consumers** using the JSON-LD output who need to understand what schema.org types represent Interlisp bibliography entries
+
+To recommend a new title for inclusion in the bibliography, please see the [Interlisp Bibliography Intake Guide](history/bibliography/intake).  It explains how to recommend new bibliography items and provides a link to the appropriate form.
+
+The mapping chain flows through three formats:
+
+```
+Zotero Item Type → CSL JSON Type → Schema.org @type
+```
+
+Each format serves a different purpose:
+
+- **Zotero Item Type**: Classification used within Zotero (the source system)
+- **CSL JSON Type**: Standardized format for citation metadata (intermediate processing format)
+- **Schema.org @type**: Structured data markup for web search engines and semantic web consumers (output format)
+
+A Zotero item type maps to exactly one CSL JSON type. The CSL JSON type is what the rest of the pipeline sees, and it is the value recorded as `item_type` in each bibliography entry's front matter.
+
+---
+
+## Type Reference Table
+
+| Zotero Item Type | CSL Type | Schema.org @type | additionalType | genre | Notes |
+|---|---|---|---|---|---|
+| **Journal Article** | `article-journal` | `ScholarlyArticle` | — | — | Peer-reviewed journal articles, conference proceedings in journals |
+| **Magazine Article** | `article-magazine` | `Article` | — | — | Magazine and trade publication articles |
+| **Newspaper Article** | `article` | `Article` | — | — | Newspaper articles and news items |
+| **Book** | `book` | `Book` | — | — | Books and monographs |
+| **Book Section** | `chapter` | `Chapter` | — | — | Chapters or sections within books |
+| **Conference Paper** | `paper-conference` | `ScholarlyArticle` | — | — | Papers presented at conferences (when not in proceedings) |
+| **Report** | `report` | `Report` | the entry's report type | — | Technical reports, working papers, government documents |
+| **Thesis** | `thesis` | `Thesis` | — | — | Master's theses, dissertations, doctoral theses |
+| **Patent** | `patent` | `CreativeWork` | `"Patent"` | — | Patents and patent applications |
+| **Blog Post** | `post-weblog` | `BlogPosting` | — | — | Blog posts and web log entries |
+| **Webpage** | `webpage` | `WebPage` | — | — | General web pages and websites |
+| **Encyclopedia Entry** | `entry-encyclopedia` | `Article` | — | — | Encyclopedia and reference work entries |
+| **Film** | `motion_picture` | `VideoObject` | — | — | Films, videos, and visual media |
+| **Email** | `personal_communication` | `Message` | — | — | Personal communications, emails, letters |
+| **Computer Program** | `software` | `SoftwareSourceCode` | — | — | Software, source code repositories, applications, tools. The Type field names the programming language and maps to `programmingLanguage`; publisher, when present, maps to `publisher`. |
+| **Document** | `document` | `CreativeWork` | `"TechnicalDocumentation"` | the entry's Type, if set | Technical manuals, guides, user documentation, software docs, specifications. Publisher, when present, maps to `publisher`. |
+| **Document** with Type `Transcript` | `document` | `CreativeWork` | `"Transcript"` | `"Transcript"` | Oral history interviews, recorded interviews, media transcripts |
+| **Document** with Type `Newsletter` | `document` | `CreativeWork` | `"TechnicalDocumentation"` | `"Newsletter"` | Newsletters, bulletins, and serial publications stored as documents |
+| **Standard** | `standard` | `CreativeWork` | `"Standard"` | — (Type → `isPartOf`) | Technical standards, specifications, protocols, standards documents. The Type field names the standard's series (`isPartOf` → `CreativeWorkSeries`); the standard number maps to an `identifier`. |
+
+### Legend
+
+- **Zotero Item Type**: The classification used in Zotero's item type selector
+- **CSL Type**: The type value in CSL JSON output from Zotero API
+- **Schema.org @type**: The `@type` field in generated JSON-LD
+- **additionalType**: Optional `additionalType` field in JSON-LD for additional classification
+- **genre**: Optional `genre` field in JSON-LD for the entry's genre
+
+---
+
+## Type Categories and Use Cases
+
+### Publishing Types (Peer-Reviewed and Published)
+
+These types are for published, citable research and writing:
+
+- **Journal Article** (`article-journal`): Peer-reviewed journal papers, conference proceedings published in journals
+- **Magazine Article** (`article-magazine`): Trade magazines, popular science magazines
+- **Newspaper Article** (`article`): Newspaper articles and news publications
+- **Conference Paper** (`paper-conference`): Conference proceedings and papers
+- **Book** (`book`): Published books and monographs
+- **Book Section** (`chapter`): Chapters within books
+- **Thesis** (`thesis`): Academic theses and dissertations
+
+### Technical and Professional Types
+
+- **Report** (`report`): Technical reports, working papers, government reports, internal documentation
+- **Patent** (`patent`): Patents and patent applications
+- **Encyclopedia Entry** (`entry-encyclopedia`): Reference works and encyclopedia entries
+- **Standard** (`standard`): Technical standards, protocols, standardization documents
+
+### Documentary Types
+
+- **Document** (`document`): Broad category for technical documentation, user guides, specifications, manuals
+  - The Zotero **Type** field distinguishes subcategories within this item type. For an oral-history interview — a written document recording a conversation between an interviewer and an interviewee — set the Type field to `Transcript`. Its schema.org mapping is `additionalType: "Transcript"` with `genre: "Transcript"`.
+  - Other Type values are carried through to the Schema.org `genre` property. For example, a document with a Type of `Newsletter` maps to `genre: "Newsletter"`.
+- **Standard** (`standard`): The Zotero **Type** field names the standard's series or designation (for example, `Xerox System Integration Standard`), which maps to `isPartOf` → `CreativeWorkSeries`. The entry's `@type` is `CreativeWork` with `additionalType: "Standard"`, and the standard number is emitted as a `PropertyValue` identifier.
+
+### Digital and Media Types
+
+- **Webpage** (`webpage`): Web pages, online resources, digital articles
+- **Blog Post** (`post-weblog`): Blog entries and web logs
+- **Film** (`motion_picture`): Videos, films, recorded presentations, documentaries
+- **Computer Program** (`software`): Software repositories, source code, applications, development tools. The Zotero **Type** field names the programming language, which maps to the Schema.org `programmingLanguage` property.
+
+### Personal Communication
+
+- **Email** (`personal_communication`): Emails, letters, personal correspondence
+
+---
+
+## For Content Creators
+
+### Choosing the Right Type
+
+When adding an entry to the Interlisp Zotero group:
+
+1. **Select the most specific Zotero item type** that matches your entry
+   - Use `Journal Article` for peer-reviewed journal papers
+   - Use `Conference Paper` for conference proceedings (not in a journal)
+   - Use `Report` for technical reports and internal documentation
+   - Use `Document` only when no other type fits (broad catch-all)
+
+2. **Fill in the Type field** for the item types that use it
+   - `Document`: the kind of document. Use `Transcript` for an oral-history interview.
+   - `Standard`: the standard's series or designation
+   - `Computer Program`: the programming language
+
+3. **Verify your schema.org mapping** by checking this reference table
+
+### Example: Choosing Between Report and Document
+
+| Aspect | Report | Document |
+|--------|--------|----------|
+| **Best for** | Published technical reports, working papers | User manuals, specifications, guides |
+| **Publisher info available** | Usually yes | Often no |
+| **Formal publication** | Yes, often | Rarely |
+| **Organization issuing** | Clear (government, company, academic) | May be vague |
+| **Use when** | Official documentation with publication date | In-house documentation, unpublished guides |
+
+---
+
+## For Developers
+
+### Processing Pipeline
+
+The bibliography processing pipeline:
+
+1. **Fetches data** from the Zotero API (v3) for group 2914042
+2. **Extracts** the Zotero itemType and the CSL JSON type
+3. **Generates markdown frontmatter** via `bibSplit.pl` (uses the CSL type)
+4. **Generates JSON-LD** via the `bibliography-json-ld.html` partial (maps the CSL type to schema.org)
+5. **Publishes** markdown files and embedded JSON-LD to the website
+
+The CSL JSON type is the entry's type of record. The Zotero **Type** field is a separate thing: it is a generic classification field whose meaning depends on the item type, and Zotero's CSL export maps it to the CSL `genre` field. It never names the entry's type.
+
+### Current Type Handling
+
+See `layouts/_partials/bibliography-json-ld.html` for the active type mappings:
+
+```go
+{{- if eq $itemType "article-journal" -}}{{- $schemaType = "ScholarlyArticle" -}}
+{{- else if eq $itemType "book" -}}{{- $schemaType = "Book" -}}
+{{- else if eq $itemType "chapter" -}}{{- $schemaType = "Chapter" -}}
+// ... etc
+```
+
+### Adding New Types
+
+To add a type:
+
+1. Add a type handler in `scripts/bibSplit.pl` (the `$type` branch chain, line ~146) so the pipeline emits it
+2. Add the @type mapping in `layouts/_partials/bibliography-json-ld.html` (line ~25)
+3. Add an `additionalType` assignment if the type needs further refinement (line ~124)
+4. Add a test fixture under `tests/fixtures/bibliography/` and register its slug in `tests/conftest.py`
+5. Add a test class in `tests/test_bibliography_jsonld.py`
+6. Update this reference table and `docs/bibliography-schema.md`
+
+### Known Issues
+
+See `docs/bibliography-schema.md` for known issues and limitations in bibliography processing.
+
+---
+
+## Schema.org Vocabulary Notes
+
+### CreativeWork Hierarchy
+
+Several types map to the generic `CreativeWork` in schema.org:
+
+- `patent` → `CreativeWork` (with `additionalType: "Patent"`)
+- `document` → `CreativeWork` (with `additionalType: "TechnicalDocumentation"`)
+- `document` with a `Transcript` type → `CreativeWork` (with `additionalType: "Transcript"`)
+- `standard` → `CreativeWork` (with `additionalType: "Standard"`)
+
+This hierarchy allows search engines to understand that these are creative works while providing additional specificity through `additionalType`.
+
+### Why Not More Specific Types?
+
+Schema.org has no native types for:
+- Generic "documents"
+- Transcripts (unless linked to media)
+- Technical standards (as standalone documents)
+- General software (though `SoftwareSourceCode` is specific)
+
+Using `CreativeWork` with `additionalType` is the recommended schema.org pattern for these cases. Schema.org's `additionalType` documentation permits text values "sparingly, for cases where useful information can be added without their being an appropriate schema to reference", with the class label following the schema.org style guide.
+
+### The genre Property
+
+Schema.org's `genre` property ("Genre of the creative work") is the natural home for an entry's genre, and it is where the Zotero **Type** field is carried for `Document` entries. Both are used together for transcripts: `additionalType` states that the work is a transcript, while `genre` records the classification in schema.org's first-class slot.
+
+The Zotero **Type** field is generic, so its meaning depends on the item type, and so does its schema.org destination:
+
+| Zotero item type | Type field contains | Expressed in schema.org as |
+|---|---|---|
+| `Document` | the kind of document, e.g. `Transcript` | `genre`; `additionalType` is always set (`TechnicalDocumentation`, or `Transcript`) |
+| `Standard` | the standard's series or designation | `isPartOf` → `CreativeWorkSeries` |
+| `Computer Program` | the programming language | `programmingLanguage` |
+| `Report` | the report type | `additionalType` |
+| `Thesis` | the degree type | not currently mapped |
+| `Webpage` | the kind of site, e.g. `Blog` | not currently mapped (`website_type` is frontmatter only) |
+
+---
+
+## Troubleshooting
+
+### Entry Not Appearing with Expected Type
+
+1. **Check CSL JSON output**: Run `scripts/update_bibliography.sh` and examine the raw JSON for your entry's `type` field
+2. **Verify mapping**: Check that the CSL type appears in the table above
+3. **Check the Type field**: The CSL type is authoritative. A `type` value that looks like a genre word (for example `Transcript` or `Newsletter`) rather than a CSL type means the merge-precedence defect in `bib-fns.jq` (`semiflatten`) has regressed — the Zotero **Type** field must never overwrite the CSL type
+4. **Report missing type**: If type is in CSL JSON but not in table, file an issue
+
+### JSON-LD Validation Errors
+
+1. **Use Google Structured Data Testing Tool** to validate: https://search.google.com/structured-data/testing-tool
+2. **Check schema.org** for @type requirements: https://schema.org/
+3. **Review generated HTML** in browser DevTools (JSON-LD is embedded in `<script type="application/ld+json">`)
+
+---
+
+## References
+
+- [Zotero Item Types](https://www.zotero.org/support/kb/item_types)
+- [CSL JSON Schema](https://citeproc-js.readthedocs.io/en/latest/csl-json/markup.html)
+- [Schema.org Types](https://schema.org/Thing)
+- [Schema.org CreativeWork](https://schema.org/CreativeWork)
+- [Schema.org additionalType](https://schema.org/additionalType)
+- [Schema.org genre](https://schema.org/genre)
+- [Structured Data Best Practices](https://developers.google.com/search/docs/beginner/structured-data)
+
+## Related Documentation
+
+- [Interlisp Bibliography Intake Guide](../intake/) – How to recommend a new bibliography item
+- `docs/bibliography-schema.md` – Technical details of the bibliography processing system (repository source)

layouts/_partials/bibliography-json-ld.html

@@ -35,6 +35,8 @@
 {{- else if eq $itemType "motion_picture"         -}}{{- $schemaType = "VideoObject"        -}}
 {{- else if eq $itemType "entry-encyclopedia"     -}}{{- $schemaType = "Article"            -}}
 {{- else if eq $itemType "personal_communication" -}}{{- $schemaType = "Message"            -}}
+{{- else if eq $itemType "document"               -}}{{- $schemaType = "CreativeWork"       -}}
+{{- else if eq $itemType "standard"               -}}{{- $schemaType = "CreativeWork"       -}}
 {{- else if eq $itemType "software"               -}}{{- $schemaType = "SoftwareSourceCode" -}}
 {{- end -}}
 
@@ -142,6 +144,39 @@
   {{- end -}}
 {{- end -}}
 
+{{- /* ── Document / Standard / Software: genre-driven refinement ─────── */ -}}
+{{- $genre := .Params.genre | default "" | strings.TrimSpace | lower -}}
+
+{{- if eq $itemType "document" -}}
+  {{- if eq $genre "transcript" -}}
+    {{- $ld = merge $ld (dict "additionalType" "Transcript" "genre" "Transcript") -}}
+  {{- else -}}
+    {{- $ld = merge $ld (dict "additionalType" "TechnicalDocumentation") -}}
+    {{- with .Params.genre -}}{{- $ld = merge $ld (dict "genre" .) -}}{{- end -}}
+  {{- end -}}
+{{- end -}}
+
+{{- if eq $itemType "standard" -}}
+  {{- $ld = merge $ld (dict "additionalType" "Standard") -}}
+  {{- with .Params.genre -}}
+    {{- $ld = merge $ld (dict "isPartOf" (dict "@type" "CreativeWorkSeries" "name" .)) -}}
+  {{- end -}}
+  {{- with .Params.standard_number -}}
+    {{- $ld = merge $ld (dict "identifier" (dict "@type" "PropertyValue" "propertyID" "standard-number" "value" .)) -}}
+  {{- end -}}
+{{- end -}}
+
+{{- if eq $itemType "software" -}}
+  {{- with .Params.genre -}}{{- $ld = merge $ld (dict "programmingLanguage" .) -}}{{- end -}}
+{{- end -}}
+
+{{- /* ── Publisher for types that carry it ───────────────────────────── */ -}}
+{{- if or (eq $itemType "document") (eq $itemType "standard") (eq $itemType "software") -}}
+  {{- with .Params.publisher -}}
+    {{- $ld = merge $ld (dict "publisher" (dict "@type" "Organization" "name" .)) -}}
+  {{- end -}}
+{{- end -}}
+
 {{- /* ── Journal / Magazine article ────────────────────────────────────── */ -}}
 {{- if or (eq $itemType "article-journal") (eq $itemType "article-magazine") -}}
   {{- with .Params.publication_title -}}

layouts/bibliography/single.html

@@ -94,10 +94,10 @@ <h1>{{ .Title }}</h1>
         {{ with .Params.publisher }}
             <p><strong>Publisher:</strong> {{ . }}</p>
         {{ end }}
-        {{ with .Params.reportType }}
+        {{ with .Params.report_type }}
             <p><strong>Report Type:</strong> {{ . }}</p>
         {{ end }}
-        {{ with .Params.reportNumber }}
+        {{ with .Params.report_number }}
             <p><strong>Report Number:</strong> {{ . }}</p>
         {{ end }}
     {{- end -}}
@@ -187,6 +187,42 @@ <h1>{{ .Title }}</h1>
             <p><strong>Encyclopedia Title:</strong> {{ . }}</p>
         {{ end }}
     {{- end -}}
+    {{- if eq .Params.item_type "document" }}
+        {{ with .Params.genre }}
+            <p><strong>Document Type:</strong> {{ . }}</p>
+        {{ end }}
+        {{ with .Params.publisher }}
+            <p><strong>Publisher:</strong> {{ . }}</p>
+        {{ end }}
+        {{ with .Params.place }}
+            <p><strong>Place:</strong> {{ . }}</p>
+        {{ end }}
+    {{- end -}}
+    {{- if eq .Params.item_type "standard" }}
+        {{ with .Params.genre }}
+            <p><strong>Series:</strong> {{ . }}</p>
+        {{ end }}
+        {{ with .Params.standard_number }}
+            <p><strong>Standard Number:</strong> {{ . }}</p>
+        {{ end }}
+        {{ with .Params.standard_version }}
+            <p><strong>Version:</strong> {{ . }}</p>
+        {{ end }}
+        {{ with .Params.publisher }}
+            <p><strong>Publisher:</strong> {{ . }}</p>
+        {{ end }}
+        {{ with .Params.place }}
+            <p><strong>Place:</strong> {{ . }}</p>
+        {{ end }}
+    {{- end -}}
+    {{- if eq .Params.item_type "software" }}
+        {{ with .Params.genre }}
+            <p><strong>Programming Language:</strong> {{ . }}</p>
+        {{ end }}
+        {{ with .Params.publisher }}
+            <p><strong>Publisher:</strong> {{ . }}</p>
+        {{ end }}
+    {{- end -}}
   </div>
   <p>
     {{- if .Params.url_source -}}