timelink.kleio package

Subpackages

Submodules

timelink.kleio.importer module

timelink.kleio.importer.import_from_xml(filespec: str | Path, session: Session, options: dict | None = None) → dict[source]

Import data from file or url into a timelink-mhk database.

The data file must be a XML file in the format exported by the kleio translator.

Parameters:

  • filespec (str,Path) – a file path, URL of a data file or path in a Kleio Server.

  • session – a database session from TimelinkDatabase.session()

  • options (dict) –

    a dictionnary with options

    • ’return_stats’: if True import stats will be returned

    • ’kleio_url’: the url of kleio server;

    • ’kleio_token’: the authorization token for the kleio server.

    • ’mode’: the mode of the import, either ‘TL’(Timelink) or ‘MHK’

  • from (If kleio_url and kleio_token are specified the data will be fetched)

  • file (a KleioServer and the filespec should contain the "xml_path" of the)

  • server (in the)

  • autentication (the kleio_token will be inserted in the header for)

Returns:

If stats is True in options a dict with statistical information will be returned.

  • ’datetime’: the time stamp of the start of the import

  • ’machine’: local machine name

  • ’file’: path to imported file or url

  • ’import_time_seconds’: elapsed time during import

  • ’entities_processed’: number of entities processed

  • ’entity_rate’: number of entities processed per second

  • ’person_rate’: number of persons (entities of class ‘person’)

  • ’nerrors’: number of errors during import

  • ’errors’: list of error messages

Examples

Returned statistical information when stats=True

{
'datetime': '2022-01-02 18:11:12',
'machine': 'joaquims-mbpr.local',
'file': 'https://...b1685.xml',
'import_time_seconds': 7.022288084030151,
'entities_processed': 747,
'entity_rate': 106.37558457603042,
'person_rate': 27.483919441999827
'nerrors': 0
'errors': []
}

TODO: should use https when the kleio_url not local.

timelink.kleio.kleio_server module

Interface to Kleio Server

class timelink.kleio.kleio_server.KleioServer(container: Container | None = None, url: str | None = None, token: str | None = None, kleio_home: str | None = None)[source]

This class interfaces to a Kleio server through its JSON-RPC api. It also provides convenience methods to start a server in Docker locally.

This class is not intended to be used directly. Use KleioServer.start() and KleioServer.attach() to create instances of KleioServer.

Parameters:

  • container (docker.models.containers.Container) – runing kleio server container

  • url (str) – kleio server url if running in a different machine (container=None)

  • token (str) – kleio server token if running in a different machine (container=None)

  • kleio_home (str) – kleio server home directory. If None and container is not None then kleio_home is obtained from the container. If not none

Not to be used directly.

See KleioServer.start() and KleioServer.attach()

To start a kleio server locally in docker use KleioServer.start()

To attach to a running kleio server, local or remote, use KleioServer.attach()

Parameters:

  • container (docker.models.containers.Container) – runing kleio server container; if container is None, then url, token and kleio_home must be provided.

  • url (str) – kleio server url if running in a different machine (container=None)

  • token (str) – kleio server token if running in a different machine (container=None)

  • kleio_home (str) – kleio server home directory. If None and container is not None then kleio_home is obtained from the container. If not none

static attach(url: str, token: str, kleio_home: str | None = None)[source]

Attach to a already running Kleio Server.

Use this either to attach to a running server outside docker (for instance in a Prolog session) or to a server running in another machine (should use htpps url in that case)

Parameters:

  • url (str) – kleio server url

  • token (str) – kleio server token

  • kleio_home (str, optional) – kleio server home directory

Returns:

KleioServer object

Return type:

KleioServer

TODO #33: from urllib.parse import urlparse

url = ‘http://user:token@localhost:8088/path_to_kleio_home’ parsed_url = urlparse(url)

print(‘scheme:’, parsed_url.scheme) print(‘netloc:’, parsed_url.netloc) print(‘path:’, parsed_url.path) print(‘username:’, parsed_url.username) print(‘password:’, parsed_url.password) print(‘hostname:’, parsed_url.hostname) print(‘port:’, parsed_url.port)

call(method: str, params: dict, token: str | None = None, timeout=60)[source]

Call kleio server API

Parameters:

  • method (str) – kleio server API method

  • params (dict) – kleio server API method parameters

  • token (str, optional) – kleio server token; defaults to None -> use admin token

  • timeout (int, optional) – timeout in seconds; defaults to 60.

Returns:

kleio server API response

Return type:

dict

container: Container

kleio server container

static find_local_kleio_home(path: str | None = None)[source]

Find kleio home directory.

Kleio home directory is the directory where Kleio Server finds sources and auxiliary files like structures, mappings and inferences.

It can be in the current directory, parent directory, or tests directory. It can be named “kleio-home”, “timelink-home”, or “mhk-home”.

A special case is when the current directory is “notebooks”. In this case, kleio-home is assumed to be the parent directory of “notebooks” and thus set up as the timelink-project template.

Parameters:

path (str, optional) – path to start searching from; defaults to None -> current directory.

Returns:

kleio home directory

Return type:

str

generate_token(user: str, info: TokenInfo)[source]

Generate a token for a user

Parameters:

  • user (str) – user to generate token for

  • info (TokenInfo) – token information

Returns:

kleio server API response

Return type:

dict

get_container()[source]

Get the kleio server container

Returns:

kleio server container

Return type:

docker.models.containers.Container

get_home_page(token=None) → str[source]

Get home page from Kleio Server

Parameters:

server_url (str) – url of the content in the Kleio Server

get_kleio_home()[source]

Get the kleio server home directory

Returns:

kleio server home directory

Return type:

str

get_logs(**kwargs)[source]

Get the logs of the kleio server container

Parameters:

  • **kwargs – arguments to pass to docker container logs:

  • tail (str) – number of lines to show from the end of the logs

  • since (str) – show logs since a datetime timestamp integer seconds

  • until (str) – show logs until a datetime timestamp integer seconds

  • timestamps (bool) – show timestamps

  • follow (bool) – follow log output

Returns:

kleio server logs

Return type:

str

get_report(rpt_url: str, token=None) → str[source]

Get report from kleio server

Parameters:

rpt_url (str) – url of the report in the Kleio Server

Returns:

kleio server API response

Return type:

dict

static get_server(kleio_home: str | None = None, kleio_version='latest')[source]

Check if a kleio server is running in docker mapped to a given kleio home directory.

If yes return a KleioServer object, otherwise return None If a specific version is required, it can be specified in kleio_version

Parameters:

kleio_home (str, optional) – kleio home directory; defaults to None -> any kleio home.

Returns:

KleioServer object or None

Return type:

KleioServer or None

get_sources(path: str, recurse: str, token=None)[source]

Get sources from kleio server

Parameters:

  • path (str) – path to the directory in sources

  • recurse (str) – if “yes” recurse in subdirectories

Returns:

kleio server API response

Return type:

dict

get_token()[source]

Get the kleio server token

Returns:

kleio server token

Return type:

str

get_translations(path: str, recurse: str = True, status: str | None = None, token: str | None = None) → List[KleioFile][source]

Get translations from kleio server

Parameters:

  • path (str) – Path to the directory in sources.

  • recurse (str) – If “yes”, recurse in subdirectories.

  • status (str, optional) – Filter by translation status. Options include: V = valid translations T = need translation (source more recent than translation) E = translation with errors W = translation with warnings P = translation being processed Q = file queued for translation

  • token (str, optional) – Kleio server token.

Returns:

List of KleioFile objects.

Return type:

list[KleioFile]

get_url()[source]

Get the kleio server url

Returns:

kleio server url

Return type:

str

get_url_content(server_url: str, token=None, timeout=30) → str[source]

Get content from Kleio Server

Parameters:

  • server_url (str) – url of the content in the Kleio Server

  • token (str, optional) – Kleio server token; defaults to None -> use admin token.

  • timeout (int, optional) – timeout in seconds; defaults to 30.

host: str

kleio server host

invalidate_user(user: str)[source]

Invalidate a user

Parameters:

user (str) – user to invalidate

Returns:

kleio server API response

Return type:

dict

static is_server_running(kleio_home: str | None = None, kleio_version='latest')[source]

Check if a kleio server is running in docker mapped to a given kleio home directory.

Parameters:

  • kleio_home (str, optional) – kleio home directory; defaults to None -> any kleio home.

  • kleio_version (str, optional) – kleio server version; defaults to “latest” if not specified only checks latest

Return True of False

Parameters:

kleio_home (str, optional) – kleio home directory; defaults to None -> any kleio home.

Returns:

True if server is running, False otherwise

Return type:

bool

kleio_admin_token: str

kleio server admin token

kleio_home: str

kleio server home directory

static make_token()[source]

Get the kleio server token from environment or generate a new one if not set.

Returns:

kleio server token

Return type:

str

static start(kleio_image: str = 'timelinkserver/kleio-server', kleio_version: str | None = 'latest', kleio_home: str | None = None, kleio_admin_token: str | None = None, kleio_server_port='8088', kleio_external_port=None, kleio_server_workers='3', kleio_idle_timeout=900, kleio_conf_dir=None, kleio_source_dir=None, kleio_stru_dir=None, kleio_token_db=None, kleio_default_stru=None, kleio_debug=None, consistency: str = 'cached', port: int | None = None, update: bool = False, reuse: bool = True, stop_duplicates: bool = False)[source]

Starts a kleio server in docker

Parameters:

  • kleio_image (str) – kleio server image, defaults to “timelinkserver/kleio-server”

  • kleio_version (str, optional) – kleio-server image version, defaults to “latest”

  • kleio_home (str, optional) – kleio home directory, defaults to None -> current directory

  • kleio_token (str, optional) – kleio server admin token, defaults to None -> generate a random token

  • kleio_server_port (str, optional) – kleio server port (in the container), defaults to “8088”.

  • kleio_external_port (str, optional) – kleio server external port (in the host), defaults to “8089”.

  • kleio_server_workers (str, optional) – number kleio server workers, defaults to “3”.

  • kleio_idle_timeout (int, optional) – kleio server idle timeout, defaults to 900.

  • kleio_conf_dir (str, optional) – kleio server configuration directory, defaults to None.

  • kleio_source_dir (str, optional) – kleio server sources directory, defaults to None.

  • kleio_stru_dir (str, optional) – kleio server structures directory, defaults to None.

  • kleio_token_db (str, optional) – kleio server token database, defaults to None.

  • kleio_default_stru (str, optional) – kleio server default structure, defaults to None.

  • kleio_debug (str, optional) – kleio server debug level, defaults to None.

  • consistency (str, optional) – consistency of the volume mount, defaults to “cached”

  • port (int, optional) – port to map to 8088, defaults to None -> find a free port starting at 8088

  • update (bool, optional) – update kleio server image, defaults to False

  • reuse (bool, optional) – if True, reuse an existing kleio server container with same keio_home, defaults to True.

  • stop_duplicates (bool, optional) – if True, stop and remove duplicate containers with same kleio_home, defaults to False.

Returns:

KleioServer object

Return type:

KleioServer

stop()[source]

Stop the kleio server container

translate(path: str, recurse: str = 'yes', spawn: str = 'yes', token=None)[source]

Translate sources from kleio server

Parameters:

  • path (str) – path to the directory in sources

  • recurse (str, optional) – if “yes” recurse in subdirectories

  • spawn (str, optional) – if “yes” spawn a translation process for each file

Returns:

kleio server API response

Return type:

dict

translation_clean(path: str, recurse: str)[source]

clean translations from kleio server

Removes translation results from kleio server.

Parameters:

  • path (str) – path to the directory in sources

  • recurse (str) – if “yes” recurse in subdirectories

Returns:

kleio server API response

Return type:

dict

url: str

kleio server url

exception timelink.kleio.kleio_server.KleioServerDockerException[source]

Problem with docker

exception timelink.kleio.kleio_server.KleioServerException[source]
exception timelink.kleio.kleio_server.KleioServerForbidenException[source]

Corresponds to json_rpc error code -32006 or HTTP 403

timelink.kleio.kleio_server.find_local_kleio_home(path: str | None = None)[source]

Find kleio home directory.

Kleio home directory is the directory where Kleio Server finds sources and auxiliary files like structures, mappings and inferences.

It can be in the current directory, parent directory, or tests directory. It can be named “kleio-home”, “timelink-home”, or “mhk-home”.

Special cases:
  • if the current directory is “notebooks”, kleio-home is assumed that Kleio home

    to be the parent directory of “notebooks”

  • if there is a “tests” subdirectory, kleio-home is searched in childs of “tests”

timelink.kleio.kleio_server.get_kserver_container(kleio_home: str | None = None, kleio_version='latest', stop_duplicates=False)[source]

Check if a kleio server is running in docker, possibly mapped to a given kleio home directory.

Parameters:

  • kleio_home (str, optional) – kleio home directory; defaults to None -> any kleio home.

  • kleio_version (str, optional) – kleio server version; defaults to “latest” if not specified only checks latest

  • stop_duplicates (bool, optional) – if True, stop and remove duplicate containers with same kleio_home; defaults to False.

Returns:

the Kleio server container

Return type:

docker.models.containers.Container

timelink.kleio.kleio_server.get_kserver_container_list(kleio_version: str | None = None) → None | List[Container][source]

Get the Kleio server containers currently running in docker

Running containers are inspected to detect those based on images with “kleio-server” on a tag.

Parameters:

kleio_version (str, optional) – kleio server version; defaults to “latest”

Returns:

the Kleio server container

Return type:

docker.models.containers.Container

timelink.kleio.kleio_server.get_kserver_home(container: Container | None = None, container_number: int = 0, kleio_version='latest')[source]

Get the kleio server home directory

Parameters:

  • container (docker.models.containers.Container, optional) – kleio server container; defaults to None -> get by number.

  • container_number (int, optional) – container number. Defaults to 0.

  • kleio_version (str, optional) – kleio server version; defaults to “latest”

Returns the volume mapped to /kleio-home in the kleio server container

timelink.kleio.kleio_server.get_kserver_token(container: Container | None = None, container_number: int = 0, kleio_version='latest') → str[source]

Get the Kleio server container admin token

Parameters:

  • container (docker.models.containers.Container, optional) – kleio server container; defaults to None -> get the first container.

  • container_number (int, optional) – container number. Defaults to 0.

  • kleio_version (str, optional) – kleio server version; defaults to “latest”

Returns:

the kleio server container token

Return type:

str

timelink.kleio.kleio_server.is_docker_running()[source]

Check if docker is running

timelink.kleio.kleio_server.list_kleio_server_containers(kleio_version: str | None = None) → List[Tuple[str, str, int, Container]][source]

List running kleio server containers

Parameters:

kleio_version (str, optional) – kleio server version; defaults to None -> any version.

Returns:

a tuple (name, kleio_home, port, token, container) for each container

timelink.kleio.kleio_server.random_token(length=32)[source]

Generate a random token

timelink.kleio.kleio_server.start_kleio_server(image: str = 'timelinkserver/kleio-server', version: str | None = None, kleio_home: str | None = None, kleio_admin_token: str | None = None, kleio_server_port='8088', kleio_external_port=None, kleio_server_workers='3', kleio_idle_timeout=900, kleio_conf_dir=None, kleio_source_dir=None, kleio_stru_dir=None, kleio_token_db=None, kleio_default_stru=None, kleio_debug=None, consistency: str = 'cached', update: bool = False, reuse: bool = True, stop_duplicates: bool = False)[source]

Starts a kleio server in docker

Parameters:

  • image (str, optional) – kleio server image. Defaults to “time-link/kleio-server”.

  • version (str | None, optional) – kleio-server version; defaults to “latest”.

  • kleio_home (str | None, optional) – kleio home directory. Defaults to current directory.

  • kleio_admin_token (str | None, optional) – kleio server token; if None -> generate a random token.

  • kleio_server_port (str, optional) – kleio server port (in the container). Defaults to “8088”.

  • kleio_external_port (str, optional) – kleio server external port (in the host).

  • kleio_server_workers (str, optional) – number kleio server workers. Defaults to “3”.

  • kleio_idle_timeout (int, optional) – kleio server idle timeout. Defaults to 900.

  • kleio_conf_dir (str, optional) – kleio server configuration directory. Defaults to None.

  • kleio_source_dir (str, optional) – kleio server sources directory. Defaults to None.

  • kleio_stru_dir (str, optional) – kleio server structures directory. Defaults to None.

  • kleio_token_db (str, optional) – kleio server token database. Defaults to None.

  • kleio_default_stru (str, optional) – kleio server default structure. Defaults to None.

  • kleio_debug (str, optional) – kleio server debug level. Defaults to None.

  • consistency (str, optional) – consistency of the volume mount. Defaults to “cached”.

  • port (int, optional) – port to map to 8088, if none find a free port starting at 8088.

  • update (bool, optional) – update kleio server image. Defaults to False.

  • reuse (bool, optional) – if True, reuse an existing kleio server container with same keio_home; defaults to True.

  • stop_duplicates (bool, optional) – if True, stop and remove duplicate containers

Usage:

update=True will pull the latest image; if it is newer than existing one any container running with the same kleio_home will be stopped and removed, a new container will be started with the new image and reuse=True will be ignored.

timelink.kleio.kleio_server.stop_kleio_server(container: Container | None = None)[source]

Stop kleio server

timelink.kleio.schemas module

class timelink.kleio.schemas.ApiPermissions(value)[source]

An enumeration.

class timelink.kleio.schemas.KleioFile(*, path: str, name: str, size: int, directory: str, modified: datetime, modified_iso: datetime, modified_string: str, qtime: datetime, qtime_string: str, source_url: str, status: translation_status_enum, translated: datetime | None = None, translated_string: str | None = None, errors: int | None = None, warnings: int | None = None, version: str | None = None, rpt_url: str | None = None, xml_url: str | None = None, import_status: import_status_enum | None = None, import_errors: int | None = None, import_warnings: int | None = None, import_error_rpt: str | None = None, import_warning_rpt: str | None = None, imported: datetime | None = None, imported_string: str | None = None)[source]

Represents the information about a kleio file and its translation and import status

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

needs_import()[source]

Return True if the file needs import

A file needs import if it has not been imported before

or if it has been translated again since the last import

needs_translation()[source]

Return True if the file needs translation

source_url: str

The status of the file: .. code-block:: python

V = “valid translations” T = “need translation (source more recent than translation)” E = “translation with errors” W = “translation with warnings” P = “translation being processed” Q = “file queued for translation”

class timelink.kleio.schemas.TokenInfo(*, comment: str, api: list[ApiPermissions], structures: str, sources: str)[source]

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class timelink.kleio.schemas.import_status_enum(value)[source]

An enumeration.

class timelink.kleio.schemas.translation_status_enum(value)[source]

An enumeration.

timelink.kleio.utilities module

Kleio Groups are the building blocks for transcription of historical sources.

Convert a Timelink date in the format YYYYMMDD to a Python datetime

Args: tl_date: a string representing a date in the format YYYYMMDD format: the format used to scan the date string Dates can be incomplete: 1586 or 1586-03 or 158603

Zeros can be used to register unknown values: 15860000 or 15860300

Missing information is handled in the following way: * If the year is missing returns None * If the month is missing, returns the 2nd of July of that year (middle day of the year) * If the day is missing, returns the 15th of the month (middle day of the month)

Format a timelink date YYYYMMDD and variants to nice string

timelink.kleio.utilities.get_extra_info_from_obs(obs_text: str) → tuple[str, dict][source]

Extracts the extra information from the extra_info string and returns a tuple with the cleaned string and a dictionary

Parameters:

obs_text (str) – The string with extra information (can have other text before)

Return type:

tuple[str,dict]

timelink.kleio.utilities.kleio_escape(v: str) → str[source]

Checks for Kleio special characters and quotes if needed:

>>> print(kleio_escape('normal string'))
normal string
>>> print(kleio_escape('oops we have a / in the middle'))
"oops we have a / in the middle"
timelink.kleio.utilities.quote_long_text(txt, initial_indent='    ', indent='  ', width=2048, **kwargs) → str[source]

Surround long text with triple quotes, wraps and indents lines if needed. Some of the parameters are passed on to textwrap.fill().

Sphynx style markup

Parameters:

  • txt (str) – The text to be transformed

  • initial_indent (str) – string to ident the first line of paragraphs. Default is 4 spaces. See textwrap.fill().

  • indent (str) – string to ident the wrap lines of paragraphs (after the first). Default is 2 spaces. See textwrap.fill().

  • width (int) – width of line for wrapping. See textwrap.fill().

Return type:

str

timelink.kleio.utilities.render_with_extra_info(element_name, element_value, extra_info, **kwargs) → str[source]

Renders a Kleio element with extra information

Parameters:

  • element_name (str) – The name of the element

  • element_value (str) – The value of the element from the db

  • extra_info (dict, or str (wiil be handled by get_extra_info())) – The extra information dictionary

  • kwargs – Additional parameters for quote_long_text()

Return type:

str