diff --git a/.github/workflows/test-dev.yml b/.github/workflows/test-dev.yml index 024a0c58..b573cf86 100644 --- a/.github/workflows/test-dev.yml +++ b/.github/workflows/test-dev.yml @@ -5,9 +5,11 @@ on: pull_request: branches: - main + - main-next-release push: branches: - main + - main-next-release jobs: test: diff --git a/CHANGELOG.md b/CHANGELOG.md index a5fdc449..9d322f2c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,10 @@ This release will contain a lot of new features and improvements so that a versi Not finalised: - cookie auth & its specification in TD (cookie auth branch) +## [v0.2.11] - 2025-04-25 + +- new feature - support for JSON files as backup for property values (use with `db_commit`, `db_persist` and `db_init`). Compatible only with JSON serializable properties. + ## [v0.2.10] - 2025-04-05 - bug fixes to support `class_member` properties to work with `fget`, `fset` and `fdel` methods. While using custom `fget`, `fset` and `fdel` methods for `class_member`s, diff --git a/README.md b/README.md index ceca05df..202872d4 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,9 @@ For those that understand, this package is a ZMQ/HTTP-RPC.
[![PyPI - Downloads](https://img.shields.io/pypi/dm/hololinked?label=pypi%20downloads)](https://pypistats.org/packages/hololinked) [![Conda Downloads](https://img.shields.io/conda/d/conda-forge/hololinked)](https://anaconda.org/conda-forge/hololinked) +
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.15155942.svg)](https://doi.org/10.5281/zenodo.12802841) +[![Discord](https://img.shields.io/discord/1265289049783140464?label=Discord%20Members&logo=discord)](https://discord.com/invite/kEz87zqQXh) ### To Install @@ -307,22 +310,104 @@ Here one can see the use of `instance_name` and why it turns up in the URL path. ##### NOTE - The package is under active development. Contributors welcome, please check CONTRIBUTING.md and the open issues. Some issues can also be independently dealt without much knowledge of this package. -- [example repository](https://github.com/VigneshVSV/hololinked-examples) - detailed examples for both clients and servers -- [helper GUI](https://github.com/VigneshVSV/thing-control-panel) - view & interact with your object's actions, properties and events. +- [examples repository](https://github.com/hololinked-dev/examples) - detailed examples for both clients and servers +- [helper GUI](https://github.com/hololinked-dev/thing-control-panel) - view & interact with your object's actions, properties and events. +- [live demo](https://control-panel.hololinked.dev/#https://examples.hololinked.dev/simulations/oscilloscope/resources/wot-td) - an example of an oscilloscope available for live test See a list of currently supported possibilities while using this package [below](#currently-supported). > You may use a script deployment/automation tool to remote stop and start servers, in an attempt to remotely control your hardware scripts. -### A little more about Usage +### Using APIs and Thing Descriptions The HTTP API may be autogenerated or adjusted by the user. If your plan is to develop a truly networked system, it is recommended to learn more and -use [Thing Descriptions](https://www.w3.org/TR/wot-thing-description11) to describe your hardware (This is optional and one can still use a classic HTTP client). A Thing Description will be automatically generated if absent as shown in JSON examples above or can be supplied manually. The default end point to fetch thing descriptions are:
`http(s):////resources/wot-td`
-If there are errors in generation of Thing Description -(mostly due to JSON non-complaint types), one could use:
`http(s):////resources/wot-td?ignore_errors=true` +use [Thing Descriptions](https://www.w3.org/TR/wot-thing-description11) to describe your hardware (This is optional and one can still use a classic HTTP client). A Thing Description will be automatically generated if absent as shown in JSON examples above or can be supplied manually. The default end point to fetch thing descriptions are: + +``` +http(s):////resources/wot-td +http(s):////resources/wot-td?ignore_errors=true +``` + +If there are errors in generation of Thing Description (mostly due to JSON non-complaint types), use the second endpoint which may generate at least a partial but useful Thing Description. + +### Consuming Thing Descriptions using node-wot (Javascript) + +The Thing Descriptions (TDs) can be consumed with Web of Things clients like [node-wot](https://github.com/eclipse-thingweb/node-wot). Suppose an example TD for a device instance named `spectrometer` is available at the following endpoint: + +``` +http://localhost:8000/spectrometer/resources/wot-td +``` -(client docs will be updated here next, also check official docs) +Consume this TD in a Node.js script using Node-WoT: +```js +const { Servient } = require("@node-wot/core"); +const HttpClientFactory = require("@node-wot/binding-http").HttpClientFactory; + +const servient = new Servient(); +servient.addClientFactory(new HttpClientFactory()); + +servient.start().then((WoT) => { + fetch("http://localhost:8000/spectrometer/resources/wot-td") + .then((res) => res.json()) + .then((td) => WoT.consume(td)) + .then((thing) => { + thing.readProperty("integration_time").then(async(interactionOutput) => { + console.log("Integration Time: ", await interactionOutput.value()); + }) +)}); +``` +This works with both `http://` and `https://` URLs. If you're using HTTPS, just make sure the server certificate is valid or trusted by the client. + +```js +const HttpsClientFactory = require("@node-wot/binding-http").HttpsClientFactory; +servient.addClientFactory(new HttpsClientFactory({ allowSelfSigned : true })) +``` +You can see an example [here](https://gitlab.com/hololinked/examples/clients/node-clients/phymotion-controllers-app/-/blob/main/src/App.tsx?ref_type=heads#L77). + +After consuming the TD, you can: + +
+Read Property + +`thing.readProperty("integration_time").then(async(interactionOutput) => { + console.log("Integration Time:", await interactionOutput.value()); +});` +
+
+Write Property + +`thing.writeProperty("integration_time", 2000).then(() => { + console.log("Integration Time updated"); +});` +
+
+Invoke Action + +`thing.invokeAction("connect", { serial_number: "S14155" }).then(() => { + console.log("Device connected"); +});` +
+
+Subscribe to Event + +`thing.subscribeEvent("intensity_measurement_event", async (interactionOutput) => { + console.log("Received event:", await interactionOutput.value()); +});` +
+ +Try out the above code snippets with an online example [using this TD](http://examples.hololinked.net/simulations/spectrometer/resources/wot-td). +> Note: due to reverse proxy buffering, subscribeEvent may take up to 1 minute to receive data. All other operations work fine. + +In React, the Thing Description may be fetched inside `useEffect` hook, the client passed via `useContext` hook and the individual operations can be performed in their own callbacks attached to user elements. +
+Links to Examples +For React examples using Node-WoT, refer to: + +- [example1](https://gitlab.com/hololinked/examples/clients/node-clients/phymotion-controllers-app/-/blob/main/src/App.tsx?ref_type=heads#L96) +- [example2](https://gitlab.com/hololinked/examples/clients/node-clients/phymotion-controllers-app/-/blob/main/src/components/movements.tsx?ref_type=heads#L54) +
+ ### Currently Supported - control method execution and property write with a custom finite state machine. @@ -335,15 +420,11 @@ If there are errors in generation of Thing Description - run direct ZMQ-TCP server without HTTP details - serve multiple objects with the same HTTP server, run HTTP Server & python object in separate processes or the same process -Again, please check examples or the code for explanations. Documentation is being activety improved. - -### Currently being worked - -- unit tests coverage -- separation of HTTP protocol specification like URL path and HTTP verbs from the API of properties, actions and events and move their customization completely to the HTTP server -- serve multiple things with the same server (unfortunately due to a small oversight it is currently somewhat difficult for end user to serve multiple things with the same server, although its possible. This will be fixed.) -- improving accuracy of Thing Descriptions -- cookie credentials for authentication - as a workaround until credentials are supported, use `allowed_clients` argument on HTTP server which restricts access based on remote IP supplied with the HTTP headers. This wont still help you in public networks or modified/non-standard HTTP clients. - +Again, please check examples or the code for explanations. Documentation is being actively improved. +### Contributing +See [organization info](https://github.com/hololinked-dev) for details regarding contributing to this package. There is: +- [discord group](https://discord.com/invite/kEz87zqQXh) +- [weekly meetings](https://github.com/hololinked-dev/#monthly-meetings) and +- [project planning](https://github.com/orgs/hololinked-dev/projects/4) to discuss activities around this repository. diff --git a/doc b/doc index c0c4a8d5..d4e965b5 160000 --- a/doc +++ b/doc @@ -1 +1 @@ -Subproject commit c0c4a8d5d942c9c4c360f668c1ee626c787a42b3 +Subproject commit d4e965b5ad5b8c0b88f807d031b72e76acf9cde9 diff --git a/examples b/examples index c9de52c4..aceda901 160000 --- a/examples +++ b/examples @@ -1 +1 @@ -Subproject commit c9de52c473156cf4854afa0feff9ab9af8d766ae +Subproject commit aceda901043b7da53f087b1cf46fcaaa7206f393 diff --git a/hololinked/__init__.py b/hololinked/__init__.py index 6232f7ab..5635676f 100644 --- a/hololinked/__init__.py +++ b/hololinked/__init__.py @@ -1 +1 @@ -__version__ = "0.2.10" +__version__ = "0.2.11" diff --git a/hololinked/param/parameterized.py b/hololinked/param/parameterized.py index f6f0a768..bebe7247 100644 --- a/hololinked/param/parameterized.py +++ b/hololinked/param/parameterized.py @@ -409,8 +409,10 @@ def __set__(self, obj : typing.Union['Parameterized', typing.Any], value : typin raise_ValueError("Read-only parameter cannot be set/modified.", self) value = self.validate_and_adapt(value) - - if self.class_member and obj is not self.owner: # safety check + + if not self.class_member and obj is self.owner: + raise AttributeError("Cannot set instance parameter on class") + if self.class_member and obj is not self.owner: obj = self.owner old = NotImplemented @@ -1807,8 +1809,13 @@ def __setattr__(mcs, attribute_name : str, value : typing.Any) -> None: if attribute_name != '_param_container' and attribute_name != '__%s_params__' % mcs.__name__: parameter = mcs.parameters.descriptors.get(attribute_name, None) if parameter: # and not isinstance(value, Parameter): - parameter.__set__(mcs, value) - return + try: + parameter.__set__(mcs, value) + return + except AttributeError as ex: + # raised for class attribute + if not str(ex).startswith("Cannot set instance parameter on class"): + raise ex from None return type.__setattr__(mcs, attribute_name, value) def __getattr__(mcs, attribute_name : str) -> typing.Any: diff --git a/hololinked/server/json_storage.py b/hololinked/server/json_storage.py new file mode 100644 index 00000000..a554d4e5 --- /dev/null +++ b/hololinked/server/json_storage.py @@ -0,0 +1,165 @@ +import os +import threading +from typing import Any, Dict, List, Optional, Union +from .serializers import JSONSerializer +from .property import Property +from ..param import Parameterized + + +class ThingJsonStorage: + """ + JSON-based storage engine composed within ``Thing``. Carries out property operations such as storing and + retrieving values from a plain JSON file. + + Parameters + ---------- + filename : str + Path to the JSON file to use for storage. + instance : Parameterized + The ``Thing`` instance which uses this storage. Required to read default property values when + creating missing properties. + serializer : JSONSerializer, optional + Serializer used for encoding and decoding JSON data. Defaults to an instance of ``JSONSerializer``. + """ + def __init__(self, filename: str, instance: Parameterized, serializer: Optional[Any]=None): + self.filename = filename + self.thing_instance = instance + self.instance_name = instance.instance_name + self._serializer = serializer or JSONSerializer() + self._lock = threading.RLock() + self._data = self._load() + + def _load(self) -> Dict[str, Any]: + """ + Load and decode data from the JSON file. + + Returns + ------- + value: dict + A dictionary of all stored properties. Empty if the file does not exist or cannot be decoded. + """ + if not os.path.exists(self.filename) or os.path.getsize(self.filename) == 0: + return {} + try: + with open(self.filename, 'rb') as f: + raw_bytes = f.read() + if not raw_bytes: + return {} + return self._serializer.loads(raw_bytes) + except Exception: + return {} + + def _save(self): + """ + Encode and write data to the JSON file. + """ + raw_bytes = self._serializer.dumps(self._data) + with open(self.filename, 'wb') as f: + f.write(raw_bytes) + + def get_property(self, property: Union[str, Property]) -> Any: + """ + Fetch a single property. + + Parameters + ---------- + property: str | Property + string name or descriptor object + + Returns + ------- + value: Any + property value + """ + name = property if isinstance(property, str) else property.name + if name not in self._data: + raise KeyError(f"property {name} not found in JSON storage") + with self._lock: + return self._data[name] + + def set_property(self, property: Union[str, Property], value: Any) -> None: + """ + change the value of an already existing property. + + Parameters + ---------- + property: str | Property + string name or descriptor object + value: Any + value of the property + """ + name = property if isinstance(property, str) else property.name + with self._lock: + self._data[name] = value + self._save() + + def get_properties(self, properties: Dict[Union[str, Property], Any]) -> Dict[str, Any]: + """ + get multiple properties at once. + + Parameters + ---------- + properties: List[str | Property] + string names or the descriptor of the properties as a list + + Returns + ------- + value: Dict[str, Any] + property names and values as items + """ + names = [key if isinstance(key, str) else key.name for key in properties.keys()] + with self._lock: + return {name: self._data.get(name) for name in names} + + def set_properties(self, properties: Dict[Union[str, Property], Any]) -> None: + """ + change the values of already existing few properties at once + + Parameters + ---------- + properties: Dict[str | Property, Any] + string names or the descriptor of the property and any value as dictionary pairs + """ + with self._lock: + for obj, value in properties.items(): + name = obj if isinstance(obj, str) else obj.name + self._data[name] = value + self._save() + + def get_all_properties(self) -> Dict[str, Any]: + """ + read all properties of the ``Thing`` instance. + """ + with self._lock: + return dict(self._data) + + def create_missing_properties(self, properties: Dict[str, Property], + get_missing_property_names: bool = False) -> Optional[List[str]]: + """ + create any and all missing properties of ``Thing`` instance + + Parameters + ---------- + properties: Dict[str, Property] + descriptors of the properties + + Returns + ------- + missing_props: List[str] + list of missing properties if get_missing_property_names is True + """ + missing_props = [] + with self._lock: + existing_props = self.get_all_properties() + for name, new_prop in properties.items(): + if name not in existing_props: + self._data[name] = getattr(self.thing_instance, new_prop.name) + missing_props.append(name) + self._save() + if get_missing_property_names: + return missing_props + + +__all__ = [ + ThingJsonStorage.__name__, +] diff --git a/hololinked/server/td.py b/hololinked/server/td.py index ac3d8c08..b9abfa03 100644 --- a/hololinked/server/td.py +++ b/hololinked/server/td.py @@ -1,7 +1,6 @@ import typing, inspect from dataclasses import dataclass, field -from hololinked.server.eventloop import EventLoop from .constants import JSON, JSONSerializable @@ -12,6 +11,7 @@ from .property import Property from .thing import Thing from .state_machine import StateMachine +from .eventloop import EventLoop diff --git a/hololinked/server/thing.py b/hololinked/server/thing.py index 7edbdc33..a5338faa 100644 --- a/hololinked/server/thing.py +++ b/hololinked/server/thing.py @@ -1,6 +1,7 @@ -import logging +import logging import inspect import os +import re import ssl import typing import warnings @@ -15,12 +16,13 @@ from .exceptions import BreakInnerLoop from .action import action from .dataklasses import HTTPResource, ZMQResource, build_our_temp_TD, get_organised_resources -from .utils import get_default_logger, getattr_without_descriptor_read +from .utils import get_a_filename_from_instance, get_default_logger, getattr_without_descriptor_read from .property import Property, ClassProperties from .properties import String, ClassSelector, Selector, TypedKeyMappingsConstrainedDict from .zmq_message_brokers import RPCServer, ServerTypes, EventPublisher from .state_machine import StateMachine from .events import Event +from .json_storage import ThingJsonStorage @@ -34,13 +36,13 @@ class ThingMeta(ParameterizedMetaclass): are also loaded from database at this time. One can overload ``__post_init__()`` for any operations that rely on properties values loaded from database. """ - + @classmethod def __prepare__(cls, name, bases): return TypedKeyMappingsConstrainedDict({}, type_mapping = dict( state_machine = (StateMachine, type(None)), - instance_name = String, + instance_name = String, log_level = Selector, logger = ClassSelector, logfile = String, @@ -52,13 +54,13 @@ def __prepare__(cls, name, bases): def __new__(cls, __name, __bases, __dict : TypedKeyMappingsConstrainedDict): return super().__new__(cls, __name, __bases, __dict._inner) - + def __call__(mcls, *args, **kwargs): instance = super().__call__(*args, **kwargs) instance.__post_init__() return instance - + def _create_param_container(mcs, mcs_members : dict) -> None: """ creates ``ClassProperties`` instead of ``param``'s own ``Parameters`` @@ -83,18 +85,18 @@ class Thing(Parameterized, metaclass=ThingMeta): """ __server_type__ = ServerTypes.THING # not a server, this needs to be removed. - + # local properties instance_name = String(default=None, regex=r'[A-Za-z]+[A-Za-z_0-9\-\/]*', constant=True, remote=False, doc="""Unique string identifier of the instance. This value is used for many operations, for example - creating zmq socket address, tables in databases, and to identify the instance in the HTTP Server - (http(s)://{domain and sub domain}/{instance name}). If creating a big system, instance names are recommended to be unique.""") # type: str - logger = ClassSelector(class_=logging.Logger, default=None, allow_None=True, remote=False, + logger = ClassSelector(class_=logging.Logger, default=None, allow_None=True, remote=False, doc="""logging.Logger instance to print log messages. Default logger with a IO-stream handler and network accessible handler is created if none supplied.""") # type: logging.Logger - zmq_serializer = ClassSelector(class_=(BaseSerializer, str), + zmq_serializer = ClassSelector(class_=(BaseSerializer, str), allow_None=True, default='json', remote=False, doc="""Serializer used for exchanging messages with python RPC clients. Subclass the base serializer or one of the available serializers to implement your own serialization requirements; or, register @@ -104,31 +106,31 @@ class Thing(Parameterized, metaclass=ThingMeta): doc="""Serializer used for exchanging messages with a HTTP clients, subclass JSONSerializer to implement your own JSON serialization requirements; or, register type replacements. Other types of serializers are currently not allowed for HTTP clients.""") # type: JSONSerializer - schema_validator = ClassSelector(class_=BaseSchemaValidator, default=JsonSchemaValidator, allow_None=True, + schema_validator = ClassSelector(class_=BaseSchemaValidator, default=JsonSchemaValidator, allow_None=True, remote=False, isinstance=False, doc="""Validator for JSON schema. If not supplied, a default JSON schema validator is created.""") # type: BaseSchemaValidator - + # remote properties - state = String(default=None, allow_None=True, URL_path='/state', readonly=True, observable=True, - fget=lambda self : self.state_machine.current_state if hasattr(self, 'state_machine') else None, + state = String(default=None, allow_None=True, URL_path='/state', readonly=True, observable=True, + fget=lambda self : self.state_machine.current_state if hasattr(self, 'state_machine') else None, doc="current state machine's state if state machine present, None indicates absence of state machine.") #type: typing.Optional[str] - httpserver_resources = Property(readonly=True, URL_path='/resources/http-server', - doc="object's resources exposed to HTTP client (through ``hololinked.server.HTTPServer.HTTPServer``)", + httpserver_resources = Property(readonly=True, URL_path='/resources/http-server', + doc="object's resources exposed to HTTP client (through ``hololinked.server.HTTPServer.HTTPServer``)", fget=lambda self: self._httpserver_resources ) # type: typing.Dict[str, HTTPResource] - zmq_resources = Property(readonly=True, URL_path='/resources/zmq-object-proxy', - doc="object's resources exposed to RPC client, similar to HTTP resources but differs in details.", + zmq_resources = Property(readonly=True, URL_path='/resources/zmq-object-proxy', + doc="object's resources exposed to RPC client, similar to HTTP resources but differs in details.", fget=lambda self: self._zmq_resources) # type: typing.Dict[str, ZMQResource] - gui_resources = Property(readonly=True, URL_path='/resources/portal-app', + gui_resources = Property(readonly=True, URL_path='/resources/portal-app', doc="""object's data read by hololinked-portal GUI client, similar to http_resources but differs in details.""", fget=lambda self: build_our_temp_TD(self)) # type: typing.Dict[str, typing.Any] GUI = Property(default=None, allow_None=True, URL_path='/resources/web-gui', fget = lambda self : self._gui, - doc="GUI specified here will become visible at GUI tab of hololinked-portal dashboard tool") + doc="GUI specified here will become visible at GUI tab of hololinked-portal dashboard tool") object_info = Property(doc="contains information about this object like the class name, script location etc.", URL_path='/object-info') # type: ThingInformation - - def __init__(self, *, instance_name : str, logger : typing.Optional[logging.Logger] = None, + + def __init__(self, *, instance_name : str, logger : typing.Optional[logging.Logger] = None, serializer : typing.Optional[JSONSerializer] = None, **kwargs) -> None: """ Parameters @@ -165,13 +167,19 @@ class attribute, see docs. schema validator class for JSON schema validation, not supported by ZMQ clients. db_config_file: str, optional if not using a default database, supply a JSON configuration file to create a connection. Check documentaion - of ``hololinked.server.database``. + of ``hololinked.server.database``. + use_json_file: bool, Default False + if True, a JSON file will be used as the property storage instead of a database. This value can also be + set as a class attribute. + json_filename: str, optional + If using JSON storage, this filename is used to persist property values. If not provided, a default filename + is generated based on the instance name. """ if instance_name.startswith('/'): instance_name = instance_name[1:] # Type definitions - self._owner : typing.Optional[Thing] = None + self._owner : typing.Optional[Thing] = None self._internal_fixed_attributes : typing.List[str] self._full_URL_path_prefix : str self._gui = None # filler for a future feature @@ -188,17 +196,23 @@ class attribute, see docs. zmq_serializer=zmq_serializer, http_serializer=http_serializer ) - super().__init__(instance_name=instance_name, logger=logger, + super().__init__(instance_name=instance_name, logger=logger, zmq_serializer=zmq_serializer, http_serializer=http_serializer, **kwargs) self._prepare_logger( - log_level=kwargs.get('log_level', None), + log_level=kwargs.get('log_level', None), log_file=kwargs.get('log_file', None), remote_access=kwargs.get('logger_remote_access', self.__class__.logger_remote_access if hasattr( self.__class__, 'logger_remote_access') else False) ) - self._prepare_state_machine() - self._prepare_DB(kwargs.get('use_default_db', False), kwargs.get('db_config_file', None)) + self._prepare_state_machine() + + # choose storage type, if use_json_file is True - use JSON storage, else - use database + if kwargs.get('use_json_file', + self.__class__.use_json_file if hasattr(self.__class__, 'use_json_file') else False): + self._prepare_json_storage(filename=kwargs.get('json_filename', f"{get_a_filename_from_instance(self, 'json')}")) + else: + self._prepare_DB(kwargs.get('use_default_db', False), kwargs.get('db_config_file', None)) def __post_init__(self): @@ -219,22 +233,22 @@ def _prepare_resources(self): def _prepare_logger(self, log_level : int, log_file : str, remote_access : bool = False): from .logger import RemoteAccessHandler if self.logger is None: - self.logger = get_default_logger(self.instance_name, - logging.INFO if not log_level else log_level, + self.logger = get_default_logger(self.instance_name, + logging.INFO if not log_level else log_level, None if not log_file else log_file) if remote_access: if not any(isinstance(handler, RemoteAccessHandler) for handler in self.logger.handlers): - self._remote_access_loghandler = RemoteAccessHandler(instance_name='logger', - maxlen=500, emit_interval=1, logger=self.logger) + self._remote_access_loghandler = RemoteAccessHandler(instance_name='logger', + maxlen=500, emit_interval=1, logger=self.logger) # thing has its own logger so we dont recreate one for # remote access handler self.logger.addHandler(self._remote_access_loghandler) - + if not isinstance(self, logging.Logger): for handler in self.logger.handlers: # if remote access is True or not, if a default handler is found make a variable for it anyway if isinstance(handler, RemoteAccessHandler): - self._remote_access_loghandler = handler + self._remote_access_loghandler = handler def _prepare_state_machine(self): @@ -242,48 +256,53 @@ def _prepare_state_machine(self): self.state_machine._prepare(self) self.logger.debug("setup state machine") - + def _prepare_DB(self, default_db : bool = False, config_file : str = None): - if not default_db and not config_file: + if not default_db and not config_file: self.object_info - return - # 1. create engine - self.db_engine = ThingDB(instance=self, config_file=None if default_db else config_file, - serializer=self.zmq_serializer) # type: ThingDB + return + # 1. create engine + self.db_engine = ThingDB(instance=self, config_file=None if default_db else config_file, + serializer=self.zmq_serializer) # type: ThingDB # 2. create an object metadata to be used by different types of clients object_info = self.db_engine.fetch_own_info() if object_info is not None: self._object_info = object_info # 3. enter properties to DB if not already present if self.object_info.class_name != self.__class__.__name__: - raise ValueError("Fetched instance name and class name from database not matching with the ", - "current Thing class/subclass. You might be reusing an instance name of another subclass ", - "and did not remove the old data from database. Please clean the database using database tools to ", + raise ValueError("Fetched instance name and class name from database not matching with the ", + "current Thing class/subclass. You might be reusing an instance name of another subclass ", + "and did not remove the old data from database. Please clean the database using database tools to ", "start fresh.") + def _prepare_json_storage(self, filename: str = None): + if not filename: + filename = f"{get_a_filename_from_instance(self, 'json')}" + self.db_engine = ThingJsonStorage(filename=filename, instance=self) + @object_info.getter def _get_object_info(self): if not hasattr(self, '_object_info'): self._object_info = ThingInformation( - instance_name = self.instance_name, + instance_name = self.instance_name, class_name = self.__class__.__name__, script = os.path.dirname(os.path.abspath(inspect.getfile(self.__class__))), - http_server = "USER_MANAGED", - kwargs = "USER_MANAGED", - eventloop_instance_name = "USER_MANAGED", - level = "USER_MANAGED", + http_server = "USER_MANAGED", + kwargs = "USER_MANAGED", + eventloop_instance_name = "USER_MANAGED", + level = "USER_MANAGED", level_type = "USER_MANAGED" - ) + ) return self._object_info - + @object_info.setter def _set_object_info(self, value): - self._object_info = ThingInformation(**value) + self._object_info = ThingInformation(**value) for name, thing in inspect._getmembers(self, lambda o: isinstance(o, Thing), getattr_without_descriptor_read): thing._object_info.http_server = self._object_info.http_server - - + + @property def properties(self) -> ClassProperties: """container for the property descriptors of the object.""" @@ -296,7 +315,7 @@ def _get_properties(self, **kwargs) -> typing.Dict[str, typing.Any]: skip_props = ["httpserver_resources", "zmq_resources", "gui_resources", "GUI", "object_info"] for prop_name in skip_props: if prop_name in kwargs: - raise RuntimeError("GUI, httpserver resources, RPC resources , object info etc. cannot be queried" + + raise RuntimeError("GUI, httpserver resources, RPC resources , object info etc. cannot be queried" + " using multiple property fetch.") data = {} if len(kwargs) == 0: @@ -322,9 +341,9 @@ def _get_properties(self, **kwargs) -> typing.Dict[str, typing.Any]: for rename, requested_prop in kwargs.items(): if not isinstance(self.properties[requested_prop], Property) or self.properties[requested_prop]._remote_info is None: raise AttributeError("this property is not remote accessible") - data[rename] = self.properties[requested_prop].__get__(self, type(self)) - return data - + data[rename] = self.properties[requested_prop].__get__(self, type(self)) + return data + @action(URL_path='/properties', http_method=[HTTP_METHODS.PUT, HTTP_METHODS.PATCH]) def _set_properties(self, **values : typing.Dict[str, typing.Any]) -> None: """ @@ -345,12 +364,12 @@ def _set_properties(self, **values : typing.Dict[str, typing.Any]) -> None: errors += f'{name} : {str(ex)}\n' produced_error = True if produced_error: - ex = RuntimeError("Some properties could not be set due to errors. " + + ex = RuntimeError("Some properties could not be set due to errors. " + "Check exception notes or server logs for more information.") ex.__notes__ = errors raise ex from None - @action(URL_path='/properties/db', http_method=HTTP_METHODS.GET) + @action(URL_path='/properties/db', http_method=HTTP_METHODS.GET) def _get_properties_in_db(self) -> typing.Dict[str, JSONSerializable]: """ get all properties in the database @@ -396,27 +415,27 @@ def event_publisher(self) -> EventPublisher: event publishing PUB socket owning object, valid only after ``run()`` is called, otherwise raises AttributeError. """ - return self._event_publisher - + return self._event_publisher + @event_publisher.setter def event_publisher(self, value : EventPublisher) -> None: if self._event_publisher is not None: if value is not self._event_publisher: raise AttributeError("Can set event publisher only once") - + def recusively_set_event_publisher(obj : Thing, publisher : EventPublisher) -> None: for name, evt in inspect._getmembers(obj, lambda o: isinstance(o, Event), getattr_without_descriptor_read): assert isinstance(evt, Event), "object is not an event" # above is type definition - e = evt.__get__(obj, type(obj)) - e.publisher = publisher + e = evt.__get__(obj, type(obj)) + e.publisher = publisher e._remote_info.socket_address = publisher.socket_address self.logger.info(f"registered event '{evt.friendly_name}' serving at PUB socket with address : {publisher.socket_address}") for name, subobj in inspect._getmembers(obj, lambda o: isinstance(o, Thing), getattr_without_descriptor_read): if name == '_owner': - continue + continue recusively_set_event_publisher(subobj, publisher) - obj._event_publisher = publisher + obj._event_publisher = publisher recusively_set_event_publisher(self, value) @@ -441,20 +460,20 @@ def load_properties_from_DB(self): except Exception as ex: self.logger.error(f"could not set attribute {db_prop} due to error {str(ex)}") - + @action(URL_path='/resources/postman-collection', http_method=HTTP_METHODS.GET) def get_postman_collection(self, domain_prefix : str = None): """ organised postman collection for this object """ from .api_platforms import postman_collection - return postman_collection.build(instance=self, + return postman_collection.build(instance=self, domain_prefix=domain_prefix if domain_prefix is not None else self._object_info.http_server) - + @action(URL_path='/resources/wot-td', http_method=HTTP_METHODS.GET) - def get_thing_description(self, authority : typing.Optional[str] = None, ignore_errors : bool = False): - # allow_loose_schema : typing.Optional[bool] = False): + def get_thing_description(self, authority : typing.Optional[str] = None, ignore_errors : bool = False): + # allow_loose_schema : typing.Optional[bool] = False): """ generate thing description schema of Web of Things https://www.w3.org/TR/wot-thing-description11/. one can use the node-wot as a client for the object with the generated schema @@ -482,10 +501,10 @@ def get_thing_description(self, authority : typing.Optional[str] = None, ignore_ # In other words, schema validation will always pass. from .td import ThingDescription return ThingDescription(instance=self, authority=authority or self._object_info.http_server, - allow_loose_schema=False, ignore_errors=ignore_errors).produce() #allow_loose_schema) + allow_loose_schema=False, ignore_errors=ignore_errors).produce() #allow_loose_schema) - @action(URL_path='/exit', http_method=HTTP_METHODS.POST) + @action(URL_path='/exit', http_method=HTTP_METHODS.POST) def exit(self) -> None: """ Exit the object without killing the eventloop that runs this object. If Thing was @@ -493,23 +512,23 @@ def exit(self) -> None: only be called remotely. """ if self.rpc_server is None: - return + return if self._owner is None: self.rpc_server.stop_polling() raise BreakInnerLoop # stops the inner loop of the object else: warnings.warn("call exit on the top object, composed objects cannot exit the loop.", RuntimeWarning) - + @action() def ping(self) -> None: """ping the Thing to see if it is alive""" - pass + pass - def run(self, - zmq_protocols : typing.Union[typing.Sequence[ZMQ_PROTOCOLS], - ZMQ_PROTOCOLS] = ZMQ_PROTOCOLS.IPC, + def run(self, + zmq_protocols : typing.Union[typing.Sequence[ZMQ_PROTOCOLS], + ZMQ_PROTOCOLS] = ZMQ_PROTOCOLS.IPC, # expose_eventloop : bool = False, - **kwargs + **kwargs ) -> None: """ Quick-start ``Thing`` server by creating a default eventloop & ZMQ servers. This @@ -545,28 +564,28 @@ def run(self, context = context or zmq.asyncio.Context() self.rpc_server = RPCServer( - instance_name=self.instance_name, - server_type=self.__server_type__.value, - context=context, - protocols=zmq_protocols, - zmq_serializer=self.zmq_serializer, - http_serializer=self.http_serializer, + instance_name=self.instance_name, + server_type=self.__server_type__.value, + context=context, + protocols=zmq_protocols, + zmq_serializer=self.zmq_serializer, + http_serializer=self.http_serializer, tcp_socket_address=kwargs.get('tcp_socket_address', None), logger=self.logger - ) + ) self.message_broker = self.rpc_server.inner_inproc_server - self.event_publisher = self.rpc_server.event_publisher + self.event_publisher = self.rpc_server.event_publisher from .eventloop import EventLoop self.event_loop = EventLoop( - instance_name=f'{self.instance_name}/eventloop', - things=[self], + instance_name=f'{self.instance_name}/eventloop', + things=[self], logger=self.logger, - zmq_serializer=self.zmq_serializer, - http_serializer=self.http_serializer, + zmq_serializer=self.zmq_serializer, + http_serializer=self.http_serializer, expose=False, # expose_eventloop ) - + if kwargs.get('http_server', None): from .HTTPServer import HTTPServer httpserver = kwargs.pop('http_server') @@ -579,11 +598,11 @@ def run(self, self.event_loop.run() - def run_with_http_server(self, port : int = 8080, address : str = '0.0.0.0', - # host : str = None, - allowed_clients : typing.Union[str, typing.Iterable[str]] = None, - ssl_context : ssl.SSLContext = None, # protocol_version : int = 1, - # network_interface : str = 'Ethernet', + def run_with_http_server(self, port : int = 8080, address : str = '0.0.0.0', + # host : str = None, + allowed_clients : typing.Union[str, typing.Iterable[str]] = None, + ssl_context : ssl.SSLContext = None, # protocol_version : int = 1, + # network_interface : str = 'Ethernet', **kwargs): """ Quick-start ``Thing`` server by creating a default eventloop & servers. This @@ -615,17 +634,17 @@ def run_with_http_server(self, port : int = 8080, address : str = '0.0.0.0', # send the network interface name to retrieve the IP. If a DNS server is present, you may leave this field # host: str # Host Server to subscribe to coordinate starting sequence of things & web GUI - + from .HTTPServer import HTTPServer - + http_server = HTTPServer( - [self.instance_name], logger=self.logger, serializer=self.http_serializer, + [self.instance_name], logger=self.logger, serializer=self.http_serializer, port=port, address=address, ssl_context=ssl_context, allowed_clients=allowed_clients, schema_validator=self.schema_validator, # network_interface=network_interface, **kwargs, ) - + self.run( zmq_protocols=ZMQ_PROTOCOLS.INPROC, http_server=http_server, @@ -634,6 +653,6 @@ def run_with_http_server(self, port : int = 8080, address : str = '0.0.0.0', http_server.tornado_instance.stop() - + diff --git a/hololinked/server/utils.py b/hololinked/server/utils.py index efe56b5f..a13092ed 100644 --- a/hololinked/server/utils.py +++ b/hololinked/server/utils.py @@ -224,6 +224,20 @@ def issubklass(obj, cls): return False except TypeError: return False + + +def get_a_filename_from_instance(thing: type, extension: str = 'json') -> str: + class_name = thing.__class__.__name__ + + # Remove invalid characters from the instance name + safe_instance_name = re.sub(r'[<>:"/\\|?*\x00-\x1F]+', '_', thing.instance_name) + # Collapse consecutive underscores into one + safe_instance_name = re.sub(r'_+', '_', safe_instance_name) + # Remove leading and trailing underscores + safe_instance_name = safe_instance_name.strip('_') + + filename = f"{class_name}-{safe_instance_name or '_'}.{extension}" + return filename __all__ = [ diff --git a/setup.py b/setup.py index a9e960d4..ba541157 100644 --- a/setup.py +++ b/setup.py @@ -7,7 +7,7 @@ setuptools.setup( name="hololinked", - version="0.2.10", + version="0.2.11", author="Vignesh Vaidyanathan", author_email="vignesh.vaidyanathan@hololinked.dev", description="A ZMQ-based Object Oriented RPC tool-kit for instrument control/data acquisition or controlling generic python objects.", diff --git a/tests/test_property.py b/tests/test_property.py index d17fa8a4..45e03a96 100644 --- a/tests/test_property.py +++ b/tests/test_property.py @@ -1,4 +1,5 @@ import logging, unittest, time, os +import tempfile import pydantic_core from pydantic import BaseModel from hololinked.client import ObjectProxy @@ -276,6 +277,50 @@ def test_6_pydantic_model_property(self): self.assertTrue("validation error for 'int'" in str(ex.exception)) + def test_7_json_db_operations(self): + with tempfile.NamedTemporaryFile(delete=False) as tf: + filename = tf.name + + # test db commit property + thing = TestThing(instance_name="test-db-operations", use_json_file=True, + json_filename=filename, log_level=logging.WARN) + self.assertEqual(thing.db_commit_number_prop, 0) + thing.db_commit_number_prop = 100 + self.assertEqual(thing.db_commit_number_prop, 100) + self.assertEqual(thing.db_engine.get_property('db_commit_number_prop'), 100) + + # test db persist property + self.assertEqual(thing.db_persist_selector_prop, 'a') + thing.db_persist_selector_prop = 'c' + self.assertEqual(thing.db_persist_selector_prop, 'c') + self.assertEqual(thing.db_engine.get_property('db_persist_selector_prop'), 'c') + + # test db init property + self.assertEqual(thing.db_init_int_prop, 1) + thing.db_init_int_prop = 50 + self.assertEqual(thing.db_init_int_prop, 50) + self.assertNotEqual(thing.db_engine.get_property('db_init_int_prop'), 50) + self.assertEqual(thing.db_engine.get_property('db_init_int_prop'), TestThing.db_init_int_prop.default) + del thing + + # delete thing and reload from database + thing = TestThing(instance_name="test-db-operations", use_json_file=True, + json_filename=filename, log_level=logging.WARN) + self.assertEqual(thing.db_init_int_prop, TestThing.db_init_int_prop.default) + self.assertEqual(thing.db_persist_selector_prop, 'c') + self.assertNotEqual(thing.db_commit_number_prop, 100) + self.assertEqual(thing.db_commit_number_prop, TestThing.db_commit_number_prop.default) + + # check db init prop with a different value in database apart from default + thing.db_engine.set_property('db_init_int_prop', 101) + del thing + thing = TestThing(instance_name="test-db-operations", use_json_file=True, + json_filename=filename, log_level=logging.WARN) + self.assertEqual(thing.db_init_int_prop, 101) + + os.remove(filename) + + class TestClassPropertyThing(Thing): # Simple class property with default value