add python content and ruff.toml

fixup

Author: matu3ba

assets/style.css

@@ -30,7 +30,7 @@
   --font_size_h3: 100%;
 
   --content_width: 1200px; /* same as @media screen */
-  --content_1colof2_width: calc(1200px / 2);
+  --content_1colof2_width: calc(var(--content_width) / 2);
 }
 
 body {
@@ -161,7 +161,7 @@ code.zig {
 /* .home_right { flex: 1; width: 300px; } */
 .home_left { flex: 1; }
 .home_right { flex: 1; }
-/* SHENNANIGAN: Variables break block view on Mozilla Firefox 132.0.2, so use 1200px:
+/* SHENNANIGAN: Variables break block view on Mozilla Firefox 132.0.2, so use 1200px: */
 /* @media screen and (max-width: var(--content_width)) { .home_page { display: block; } } */
 /* @media screen and (max-width: var(--content_width)) { .static_grid { display: block; } } */
 @media screen and (max-width: 1200px) { .home_page { display: block; } }

content/articles/shennanigans_in_python.smd

@@ -5,4 +5,68 @@
 .layout = "article.shtml",
 .tags = ["article", "python", "software", "design"],
 .draft = false,
----
+---
+
+[]($section.id("intro"))
+The following shennanigans were collected during my work on creating a system
+integration library and framework for hardware and software tests including
+developer tasks like worktree management, building, target deployment and
+configuration. Therefore it should be representative for things one
+might want to do in absence of better tooling. The used Python version was
+3.8.2, but most problems still persist.
+
+[]($section.id("tldr"))
+My [**tldr;**](#tldr) retrospection upfront, which you may be able to reproduce
+once you try to code long-running services, which recover from all failures,
+cleanly reset state or generally try to debug spurious problems. Please don't.
+
+Python on itself is great for prototyping up to a few hundred lines of code,
+like to quickly receive or send some json over tcp/html.
+**However**, it is unfeasible to scale, for example to use as library code.
+Changes in a leaf function can add exceptions to higher level code paths and
+handling those via exceptions for user friendly error messages, for example
+to collect context (information along multiple functions, for example from
+different combination of traversal) becomes unreasonably verbose and error
+prone. The alternative is to use C-like error handling, which requires to figure out
+all possible exceptions of Python libstd methods, which language servers
+do not support (as of 20240404).
+
+[]($section.id("list_of_shame"))
+Aside of these more fundamental limitations, here the [list of shennanigans](#list_of_shennanigans) I have run into:
+- `xml.dom.minidom` [breaks space and newlines](https://bugs.python.org/issue5752). Use `ElementTree`.
+- `.strip()` is necessary after file read, because Python automatically adds
+  `\n` and there is no way to read without newlines into a list.
+- Testing for subdictionaries with `dict` is unreadable, so such a method is missing
+[missing_dict_methods1.py]($code.asset('./missing_dict_methods1.py').language('python'))
+- `dict` has no method to check, if the fields of a dictionary are in another dictionary
+[missing_dict_methods2.py]($code.asset('./missing_dict_methods2.py').language('python'))
+- `dict` has no method to check, if all fields and values of a dictionary are in another dictionary
+[missing_dict_methods3.py]($code.asset('./missing_dict_methods3.py').language('python'))
+- Tuples and dicts are annoying to differentiate
+[tup_and_dicts.py]($code.asset('./tup_and_dicts.py').language('python'))
+- Stack trace formatting is inefficient and one can not use <code>gf</code> or <code>gF</code> vim shortcuts to jump to location
+  function to write status + trace to variable.
+[stacktrace_fmt.py]($code.asset('./stacktrace_fmt.py').language('python'))
+- Mixed double quote (`"`) and single quote (`'`) strings are invalid json
+[invalid_json.py]($code.asset('./invalid_json.py').language('python'))
+- `os.kill()` does not call registered cleanup function
+  `atexit.register(exit_cleanup)` by deamonized threads. Must store pids of
+  child processes and clean them explicitly or signal main thread via
+[signal_main_thread.py]($code.asset('./signal_main_thread.py').language('python'))
+- [Socket timeout can cause file-like readline() method to lose data](https://github.com/python/cpython/issues/51571), workaround
+  1. Read from Kernel structure and append chunk-wise to buffer from socket until stop event (via select).
+  2. After each read, try to line a line from the buffer and remove the line on success (being utf-8 etc).
+  3. On failure of reading line, continue with 1.
+  4. Teardown should read socket until being empty, if a stop was obtained.
+- Generic module annotation is not allowed and mypy has no explicit docs for
+  this. The following does not work and `module: object` is the closest we can
+  get as simple annotation.
+[module_annotation.py]($code.asset('./module_annotation.py').language('python'))
+- There are no scheduling and watchdog methods, which makes Python thread
+  scheduling very unreliable. Unlucky schedules may cause fatal delay for
+  shuffling data between daemon thread and main thread. As example, an
+  application using 1 main thread and 2 deamon threads may cause the relevant
+  deamon thread not being scheduled for 2 seconds.
+  Empirically 3 seconds work.
+- Trailing comma in dictionary or `json.dumps` generated string has
+  silent failures, for example on parsing the output as json via php.

content/articles/shennanigans_in_python/invalid_json.py

@@ -0,0 +1,14 @@
+import json
+
+
+# Dict -> str is inconsistent to json -> str, so workaround with
+# dict_asjson_lower = str(dict1).replace("'", '"')
+def combineDictsFromStr() -> None:
+  dict1 = {"t1": "val1", "t2arr": [{"t2_int": 0, "t2_str": "12.0"}], "t3int": 30}
+  dict1_str_raw = str(dict1)
+  dict1_str = dict1_str_raw.replace("'", '"')
+  dict2_str = '{"anotherone":"yes", '
+  dict2_str += '"t3int":30,"t4str":'
+  dict2_str += dict1_str + "}"
+  dict2 = json.loads(dict2_str)
+  _ = dict2

content/articles/shennanigans_in_python/missing_dict_methods1.py

@@ -0,0 +1,13 @@
+def is_subdict(small: dict, big: dict) -> bool:
+    """
+    Test, if 'small' is subdict of 'big'
+    Example: big = {'pl' : 'key1': {'key2': 'value2'}}
+    Then small = {'pl' : 'key1': {'key2': 'value2'}, 'otherkey'..} matches,
+    but small = {'pl' : 'key1': {'key2': 'value2', 'otherkey'..}}
+    or small = {'pl' : 'key1': {'key2': {'value2', 'otherkey'..}}} not.
+    """
+    # since python3.9:
+    # return big | small == big
+    # also:
+    # return {**big, **small} == big
+    return dict(big, **small) == big

content/articles/shennanigans_in_python/missing_dict_methods2.py

@@ -0,0 +1,22 @@
+def has_fieldsvals(small: dict, big: dict) -> bool:
+    """
+    Test, if 'small' has all values of of 'big'
+    Example: big = {'pl' : 'key1': {'key2': 'value2'}}
+    Then small = {'pl' : 'key1': {'key2': 'value2'}, 'otherkey'..} matches,
+    small = {'pl' : 'key1': {'key2': 'value2', 'otherkey'..}} matches,
+    and small = {'pl' : 'key1': {'key2': {'value2', 'otherkey'..}}} matches.
+    """
+    for key, value in small.items():
+        if key in big:
+            if isinstance(small[key], dict):
+                if not has_fieldsvals(small[key], big[key]):
+                    return False
+                else:
+                    return True
+            elif value != big[key]:
+                return False
+            else:
+                return True
+        else:
+            return False
+    return True

content/articles/shennanigans_in_python/missing_dict_methods3.py

@@ -0,0 +1,29 @@
+import copy
+from typing import Optional, List
+
+
+def merge_dicts(alpha: dict = {}, beta: dict = {}) -> dict:
+  """
+  Recursive merge dicts. Not multi-threading safe.
+  """
+  return _merge_dicts_aux(alpha, beta, copy.copy(alpha))
+
+
+def _merge_dicts_aux(alpha: dict = {}, beta: dict = {}, result: dict = {}, path: Optional[List[str]] = None) -> dict:
+  if path is None:
+    path = []
+  for key in beta:
+    if key not in alpha:
+      result[key] = beta[key]
+    else:
+      if isinstance(alpha[key], dict) and isinstance(beta[key], dict):
+        # key value is dict in A and B => merge the dicts
+        _merge_dicts_aux(alpha[key], beta[key], result[key], path + [str(key)])
+      elif alpha[key] == beta[key]:
+        # key value is same in A and B => ignore
+        pass
+      else:
+        # key value differs in A and B => raise error
+        err: str = f"Conflict at {'.'.join(path + [str(key)])}"
+        raise Exception(err)
+  return result

content/articles/shennanigans_in_python/module_annotation.py

@@ -0,0 +1,5 @@
+# using 'module: ModuleType' via specifying ModuleType as set of types not possible
+def check_fn(module: object) -> int:
+  if str(type(module)) != "module":
+    return 1
+  return 0

content/articles/shennanigans_in_python/signal_main_thread.py

@@ -0,0 +1,4 @@
+def signalMainThread(self) -> None:
+  pass
+  # before Python 3.10: _thread.interrupt_main()
+  # since Python 3.10: _thread.interrupt_main(signum=signal.SIGKILL)

content/articles/shennanigans_in_python/stacktrace_fmt.py

@@ -0,0 +1,5 @@
+import traceback
+
+
+def getStackTrace() -> str:
+  return repr(traceback.format_stack())

content/articles/shennanigans_in_python/tup_and_dicts.py

@@ -0,0 +1,9 @@
+# dictionary
+dict1 = {"m1": "cp", "m2": "cp"}
+# tuple
+tup1 = ({"m1": "cp", "m2": "cp"},)
+
+# at least getting the intention correct, but python is still unhelpful with error message
+dict2 = dict({"m1": "cp", "m2": "cp"})
+# tuple
+tup2 = (tuple({"m1": "cp", "m2": "cp"}),)