INA AudioDoc importer

This importer is a special case of AudioDoc format which is an output of the Whisper Algorithm. It was developed to handle ASR radio data provided by the Institut National de l’Audiovisuel (INA) of France.

INA Custom classes

INA importer classes for converting ASR data to a unified canonical format.

This module defines Issue and Audio record objects used by the INA importer to convert Automatic Speech Recognition (ASR) data into the impresso canonical format for radio-broadcast content.

class text_preparation.importers.ina.classes.INABroadcastAudioRecord(_id: str, number: int, json_filepath: str, mp3_filepath: str)

Radio-Broadcast Audio Record for INA’s ASR format.

Parameters:

_id (str) – Canonical Audio Record ID (e.g. CFCE-1900-01-02-a-r0001).
number (int) – Record number (for compatibility with other source mediums).
json_filepath (str) – Path to the JSON file containing the ASR transcript.
mp3_filepath (str) – Path to the MP3 audio file.

id

Canonical Audio Record ID (e.g. CFCE-1900-01-02-a-r0001).

Type:: str

number

Record number.

Type:: int

json_filepath

Path to the JSON file containing the ASR transcript.

Type:: str

mp3_filepath

Path to the MP3 audio file.

Type:: str

iiif_base_uri

Constructed IIIF URI for this audio record.

Type:: str

dur_in_sec

Duration of the audio record in seconds.

Type:: float | None

notes

Informational or warning messages collected during parsing.

Type:: list[str]

record_data

Audio record data according to canonical format.

Type:: dict[str, Any]

issue

Issue this audio record belongs to.

Type:: CanonicalIssue | None

Initialise the audio record and construct its canonical record data skeleton.

Parameters:

_id (str) – Canonical Audio Record ID (e.g. CFCE-1900-01-02-a-r0001).
number (int) – Record number (for compatibility with other source mediums).
json_filepath (str) – Path to the JSON file containing the ASR transcript.
mp3_filepath (str) – Path to the MP3 audio file.

add_issue(issue: CanonicalIssue) → None

Attach the parent issue to this audio record.

Parameters:: issue (CanonicalIssue) – The canonical issue this audio record belongs to.

create_iiif() → str

Create the IIIF URI for this audio record from its constituent parts.

Returns:: Constructed IIIF URI pointing to the MP3 file for this record.
Return type:: str

property json: dict

Read the JSON transcript file and return its contents as a dictionary.

Retries up to three times on I/O errors before re-raising the exception.

Returns:: Parsed contents of the ASR transcript JSON file.
Return type:: dict
Raises:: IOError – If the file cannot be read after the maximum number of retries.

parse() → None

Parse the ASR transcript and populate record_data with audio sections.

Reads the JSON transcript, extracts utterances, and builds a single audio section spanning the full broadcast. If no utterances are found, an empty section list is stored and a warning is appended to notes.

class text_preparation.importers.ina.classes.INABroadcastIssue(issue_dir: IssueDir)

Radio-Broadcast Issue for INA’s ASR format.

Wraps a single broadcast entry from the INA archive, aggregating its associated audio records and constructing the canonical issue representation.

Parameters:: issue_dir (IssueDir) – Identifying information about the issue, including the path to the data directory and provider-supplied metadata.

id

Canonical Issue ID (e.g. CFCE-1940-01-05-a).

Type:: str

edition

Lower-case letter ordering issues of the same day.

Type:: str

alias

Media title unique alias (identifier or name).

Type:: str

path

Path to the directory containing the issue’s data files.

Type:: str

date

Broadcast date of the issue.

Type:: datetime.date

metadata

Raw provider-supplied metadata from issue_dir.

Type:: dict[str, Any]

duration

Full broadcast duration as HH:MM:SS, or None if unavailable or zero.

Type:: str | None

audio_records

Audio records belonging to this issue.

Type:: list[INABroadcastAudioRecord]

issue_data

Issue data serialised to canonical format.

Type:: dict[str, Any]

Initialise the issue, discover audio records, and build canonical issue data.

Parameters:: issue_dir (IssueDir) – Identifying information about the issue, including the path to the data directory and provider-supplied metadata.

INA Detect functions

Helper functions for detecting and selecting INA ASR broadcast data to import.

text_preparation.importers.ina.detect.INAIssueDir

Lightweight data structure representing a single INA radio broadcast issue.

Can be used to locate data in the filesystem or to build canonical identifiers for the issue and its audio records.

Note

When multiple broadcasts are published on the same day, a lowercase letter indicates the edition: 'a' for the first, 'b' for the second, etc.

text_preparation.importers.ina.detect.provider

Data provider, always "INA" for this importer.

Type:: str

text_preparation.importers.ina.detect.alias

Unique broadcast title alias.

Type:: str

text_preparation.importers.ina.detect.date

Broadcast date of the issue.

Type:: datetime.date

text_preparation.importers.ina.detect.edition

Edition letter ('a', 'b', 'c', …).

Type:: str

text_preparation.importers.ina.detect.path

Path to the directory containing the issue’s data files.

Type:: str

text_preparation.importers.ina.detect.issue_metadata

Provider-supplied metadata for the issue.

Type:: dict[str, Any]

Example

>>> from datetime import date
>>> i = INAIssueDir(
...     provider='INA',
...     alias='SOC_CJ',
...     date=date(1940, 7, 22),
...     edition='a',
...     path='./SOC_CJ/1940/07/22/a',
...     issue_metadata={},
... )

text_preparation.importers.ina.detect.detect_issues(base_dir: str, alias_filter: list[str] | None = None, exclude_list: list[str] | None = None) → list[IssueDirectory]

Detect INA radio broadcast issues available for import within the filesystem.

Reads the issue index and metadata files, filters out known-faulty entries, and returns one INAIssueDir per importable issue.

Parameters:

base_dir (str) – Path to the root directory of the INA broadcast data.
alias_filter (list[str] | None) – If provided, only issues whose alias is in this list are included. Defaults to None (no filter).
exclude_list (list[str] | None) – If provided, issues whose alias appears in this list are excluded. Defaults to None (no exclusions).

Returns:

Issue instances ready for import.

Return type:

list[INAIssueDir]

text_preparation.importers.ina.detect.entry2issue(alias: str, year: str, month: str, entry: dict, base_dir: str, alias_issues: dict[str, Any]) → IssueDirectory

Convert a hierarchical JSON index entry into an INAIssueDir.

Parameters:

alias (str) – Broadcast title alias (e.g. "SOC_CJ").
year (str) – Four-digit year string (e.g. "1940").
month (str) – Two-digit month string (e.g. "07").
entry (dict) – Single issue entry from the index, containing at minimum "day" and "edition" keys (e.g. {"day": "15", "edition": "01", "local_path": ["...mp3"]}).
base_dir (str) – Absolute path to the root data directory for this alias.
alias_issues (dict[str, Any]) – Mapping of issue IDs to their provider metadata for the given alias, keyed by canonical issue ID.

Returns:

Populated named tuple for the issue.

Return type:

INAIssueDir

text_preparation.importers.ina.detect.select_issues(base_dir: str, config: dict) → list[IssueDirectory] | None

Detect and filter issues to import according to a configuration dictionary.

Behaves like detect_issues() but applies additional filtering rules specified in config. See the importer configuration documentation for the supported keys and filtering semantics.

Note

For INA, base_dir points to the original data root; each issue’s metadata contains the relative paths to its audio and transcript files.

Parameters:

base_dir (str) – Path to the root directory of the INA broadcast data.
config (dict) –
Filtering configuration. Recognised keys:
- "titles" (dict): Mapping of alias to date range(s) to include.
- "exclude_titles" (list[str]): Aliases to exclude.
- "year_only" (bool): If True, filter by year only (ignore month/day). Defaults to False.

Returns:

Filtered list of issue instances to import, or None if a required configuration key is missing.

Return type:

list[INAIssueDir] | None

INA helper functions

Helper functions used by the INA Importer.

text_preparation.importers.ina.helpers.extract_time_coords_from_elem(elem: dict, is_sseg: bool = False) → list[float] | None

Extract the time coordinates [start, duration] from a speech element.

Parameters:

elem (dict) – A word-level or speech-segment-level dict from the ASR JSON document, containing "start" / "end" timestamps and, for speech segments, a "words" list.
is_sseg (bool) – If True, treat elem as a speech segment and derive coordinates from its first and last word. If False (default), treat elem as a single word.

Returns:

A two-element list [start_time, duration] in seconds, rounded to five decimal places for the duration.

Return type:

list[float] | None

text_preparation.importers.ina.helpers.get_utterances(json_doc: list[dict]) → list[dict]

Construct utterances from consecutive speech segments sharing the same speaker.

Iterates over the speech segments in the ASR JSON document and groups consecutive segments belonging to the same speaker into a single utterance. Each utterance contains a time-code span, the speaker identifier, and the constituent speech segments with their token-level time codes.

Parameters:

json_doc (list[dict]) – Parsed ASR JSON document for one audio record. Each element is a speech-segment dict with at least "speaker" and "words" keys.

Returns:

List of utterance dicts, each containing:

"tc" (list[float]): [start_time, duration] of the utterance.

"speaker" (str): Speaker identifier.

"ss" (list[dict]): Speech segments belonging to this utterance.

Return type:

list[dict]