[pythonscripting] current marketplace version of pythonscripting next #19190

HolgerHees · 2025-08-18T12:15:13Z

Because you helped me so much last time, I'm asking you for another review :-) and in advance, sorry for the size of these pull request!

This pull request is trying to merge the current marketplace version of pythonscripting next.

Features
- OSGI Console support
  - Show additional add-on infos
  - Open an interactive python console
  - Run “pip” to install additional python modules (only if venv is enabled)
  - Update and rollback of helper lib module
  - Added Type Hint Stub Generator
- VEnv setups
  - “pip” and preinstalled “modules”
  - Support for native modules
- Preview for type hinting (autocompletion)
- Implement State data type mapping
- New rule options "runtime_measurement" & "profiler_enabled"
- Improved handling of config changes
- Improved error handling/logging for graal specific exceptions
- Refactored VFS handling
- Refactored Proxy classes
  - No manual implemented proxy classes anymore
  - All objects, including datetime objects, are now native graal foreign classes
- Cleaned and simplified PythonScriptEngine & GraalScriptEngine
  - ~20% less code => easier to understand for other people
- Initial integration tests started
- Splitted README
  1. Add-on README
  2. Helper lib README
Bugfixes
- Persistence extension handling
- Datetime <=> ZonedDateTime fixes
- Check timezone settings and print a warning on discrepancies
- Fix markedplace version deployment
- Fixes ScriptEngine reinitialisation (in Transformation Services) after Add-on updates
  - Hopefully no “Context execution was cancelled” Exceptions after Add-on updates anymore
  - An Add-on update should work now without any exception (the old one can still raise an exception on shutdown)
- Fix simple rule decorator (rule instead of rule()). Both variants works now.
- Fix ‘update install latest’ command, if a newer preview version was manuell installed
- Improved Thread and Timer handling. No need to register shutdown procedure in the lifecycleTracker anymore
- Lot of other minor cleanups
BREAKING CHANGE
- State ‘DateTimeType’ is not converted to a datetime anymore. Instead it stays as a ‘DateTimeType’, but the method getZonedDateTime returns a datetime. This leads to a much more consistent behavior.
- Metadata API changed
- Registry.safeItemName move to Item.buildSafeName
- Changed parameter of Registry.addItem
- Replace NotInitialisedException with NotFoundException for getItem, removeItem, getThing, getChannel
- openhab.Timer removed.
  - Can be replaced with default Threading or Timer implementation.
  - No need to register shutdown procedure in the lifecycleTracker anymore
  - Old timer can be found as an example

openhab-bot · 2025-08-19T19:06:21Z

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/python3-binding-discussion/161914/164

lsiepel · 2025-09-03T18:48:33Z

...nhab.automation.pythonscripting/src/main/resources/OH-INF/i18n/pythonscripting_de.properties

Revert changes to this file. The translations are managed by crowdin

i18n default translations are regenerated and committed.

Do not delete the file, revert changes to it, so it is left untouched in the PR.
If needed, you can download a fresh copy from the github repo and overwrite the file so it is in the same state as before this PR.

Sorry, but no.
You reverted changes to pythonscripting.properties while it should have been pythonscripting_de.properties

ahh, I was not aware of this file. It never exists in my branch. I reverted now both to match the version from main branch.

These "rules" might be a bit hard to understand if you're not aware of how the Crowdin integration works, but English is the "source language". It is what is found in pythonscripting.properties. If you want to change any of the texts, you should make changes in this file.

This file is then automatically uploaded to Crowdin, so that the "source strings" there are updated and translators are notified that there is something new that needs to be translated. After a while (when the translators have had time to make the translations), all the "translation files" (pythonscripting_xx.properties) are generated from the translations made at Crowdin, and these files are merged.

So, you should change pythonscripting.properties and you should not change pythonscripting_xx.properties.

...openhab.automation.pythonscripting/src/main/resources/OH-INF/i18n/pythonscripting.properties

florian-h05

Thanks for this PR, I will split the review.

Review of the README:

florian-h05 · 2025-09-26T09:04:29Z

bundles/org.openhab.automation.pythonscripting/README.md

-       return "String has " + str(len(input)) + " characters"
-   calc(input)
-   ```
+[>> openHAB Python Scripting <<](https://github.com/openhab/openhab-python/blob/main/README.md)


Would be nice to integrate this into the openHAB website where the other docs reside. We can take care of that later though, please remind me though.

The idea was to split the binding documentation from the helper lib documentation.

Both documentations together was getting longer and longer. I still gave some scripting examples in the binding documentation itself, to get an impression how it works. But the whole helper lib documentation should resist in the same repo as the helper lib itself. Before, I was always synchronizing the README's after each tiny change in the helper libs, which affects the README. Sometimes, I forgot to synchronize.

For next.openhab.org, all external documentation is synced every night, and for www.openhab.org we can also get something like this up and running.

bundles/org.openhab.automation.pythonscripting/README.md

lsiepel · 2025-09-27T09:19:04Z

Please also fix the conflicts

HolgerHees · 2025-09-27T10:11:59Z

Please also fix the conflicts

maybe I should recreate the pull request. Solving the merge conflict inside the pom is not a problem. But solving the GraalPythonScriptEngineFactory conflict is not possible, The file is renamed and completely refactored. The change in the current main branch does not make sense anymore.

I would recommend to finish the review first and then recreate the "next" branch on latest master branch and apply my changes on top of it. At the end, it completely replaces the current pythonscripting folder from main branch.

Nadahar · 2025-09-27T11:41:04Z

maybe I should recreate the pull request. Solving the merge conflict inside the pom is not a problem. But solving the GraalPythonScriptEngineFactory conflict is not possible, The file is renamed and completely refactored. The change in the current main branch does not make sense anymore.

There is no reason to create a new PR. You can force push anything to this one. The local and remote branches don't even need to have the same name, so you could create a completely new branch locally, and still push that to next on GitHub.

What I would probably do is create a new branch locally when checked out on this branch, so that you have a copy - and then, after making the copy, reset GraalPythonScriptEngineFactory to what it is in latest main, and then apply whatever changes you have made to that file manually. The rest can probably stay untouched.

HolgerHees · 2025-09-27T15:31:57Z

There is no reason to create a new PR.

I will wait until the review is done and then, maybe someone with deeper knowledge about git can help me solving this.

As I'm not a git expert... every time I tried something like this, I completely "messed up" the pull request.

That's why I'm a bit more cautious about things like that :-)

Nadahar · 2025-09-27T16:46:35Z

bundles/org.openhab.automation.pythonscripting/pom.xml

          <pmdFilter>${project.basedir}/suppressions.properties</pmdFilter>
        </configuration>
      </plugin>
+      <!-- workaround for failing clean of checked out git repo breaking the Windows build, #19250 -->


Thanks if this works, this has been quite annoying, since I've had to manually delete the target folder every time.

HolgerHees · 2025-09-29T16:11:19Z

There is no reason to create a new PR.

I tried it right now with the result I feared.

First, I added upstream repo to my git config
git remote add upstream https://github.com/openhab/openhab-addons.git

This is my config now.

>>> cat .git/config 
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = https://github.com/HolgerHees/openhab-addons.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
        remote = origin
        merge = refs/heads/main
[branch "next"]
        remote = origin
        merge = refs/heads/next
[remote "upstream"]
        url = https://github.com/openhab/openhab-addons.git
        fetch = +refs/heads/*:refs/remotes/upstream/*

After, I called git config pull.rebase false followed by a git pull upstream main

Thats the result he showed me.

Von https://github.com/openhab/openhab-addons
 * branch                  main       -> FETCH_HEAD
automatischer Merge von bundles/org.openhab.automation.pythonscripting/pom.xml
KONFLIKT (Inhalt): Merge-Konflikt in bundles/org.openhab.automation.pythonscripting/pom.xml
automatischer Merge von bundles/org.openhab.automation.pythonscripting/src/main/java/org/openhab/automation/pythonscripting/internal/scriptengine/graal/GraalPythonScriptEngineFactory.java
KONFLIKT (Inhalt): Merge-Konflikt in bundles/org.openhab.automation.pythonscripting/src/main/java/org/openhab/automation/pythonscripting/internal/scriptengine/graal/GraalPythonScriptEngineFactory.java
Automatischer Merge fehlgeschlagen; beheben Sie die Konflikte und committen Sie dann das Ergebnis.

Then I resolved the conflicts (they are as simple as I expected). But I got hundred of modified files like below.

.
.
.
.
        geändert:       itests/org.openhab.binding.ntp.tests/itest.bndrun
        geändert:       itests/org.openhab.binding.systeminfo.tests/itest.bndrun
        geändert:       itests/org.openhab.binding.systeminfo.tests/src/main/java/org/openhab/binding/systeminfo/test/SystemInfoOSGiTest.java
        geändert:       itests/org.openhab.binding.tradfri.tests/itest.bndrun
        geändert:       itests/org.openhab.persistence.mapdb.tests/itest.bndrun
        geändert:       pom.xml

Nicht zusammengeführte Pfade:
  (benutzen Sie "git add/rm <Datei>...", um die Auflösung zu markieren)
        von beiden geändert:    bundles/org.openhab.automation.pythonscripting/pom.xml
        von beiden geändert:    bundles/org.openhab.automation.pythonscripting/src/main/java/org/openhab/automation/pythonscripting/internal/scriptengine/graal/GraalPythonScriptEngineFactory.java

Whats should I do. I just want to solve the merge conflicts of the last 2 files.

@lsiepel @Nadahar maybe you have an idea?

Nadahar · 2025-09-29T16:22:22Z

After, I called git config pull.rebase false followed by a git pull upstream main

I thought that git config pull.rebase false was Git default, so I don't know why you'd need to set that unless you had previously changed it.

I never (almost - the exception is rarely with the --ff-only parameter) use git pull, it's a messy command IMO. What it does is that it combines git fetch with git merge or git rebase depending on the above parameter. I like to tell Git what I want it to do explicitly, and I generally avoid git merge as well, as merge commits are pure, evil chaos.

So, my question is what you wanted to do. To get out of the current situation, you can probably do a git merge --abort, and it should revert to where you were before you ran the "pull".

If you want to rebase on latest main, what I would do is this:

git fetch upstream
git rebase upstream/main

...then resolve any potential conflicts, after which I'd run:

git rebase --continue

...as many times as needed until all conflicts had been worked out.

As for the conflict resolution itself, I still think a proper GUI tool is needed, I know that there are people that does this in command line editors, but I just don't understand how they manage to. The conflict doesn't have to be very complicated for me to lose track of dealing with that using only the cryptic conflict information Git creates.

Then I resolved the conflicts (they are as simple as I expected). But I got hundred of modified files like below.

How did you resolve the conflicts? Did you run some script or something, since you got all the modified files? I never do that, I manually resolve each conflict... because I don't think getting everything "tainted" by some script is that much fun to clean up later.

HolgerHees · 2025-09-29T17:02:05Z

How did you resolve the conflicts?

the conflicts are not the problem. They affect only

bundles/org.openhab.automation.pythonscripting/pom.xml
bundles/org.openhab.automation.pythonscripting/src/main/java/org/openhab/automation/pythonscripting/internal/graal/GraalPythonScriptEngineFactory.java

and they are easy to solve. So no need for a GUI.

the hundred of changed files are the result the merge. I never touched them. They are marked as changes after the merge.

now I tried your recommendation regarding a rebase.

git fetch upstream
git rebase upstream/main
Resolved conflict of pom.xml & GraalPythonScriptEngineFactory.java
git rebase --continue
Resolved again a conflict in pom.xml
git rebase --continue

rebase was done.

If I try to run git push now. I got the following error

hhees@laptop-holger:~/workspace-openhab-build/github/openhab-addons> git push
To https://github.com/HolgerHees/openhab-addons.git
 ! [rejected]              next -> next (non-fast-forward)
Fehler: Fehler beim Versenden einiger Referenzen nach 'https://github.com/HolgerHees/openhab-addons.git'
Hinweis: Aktualisierungen wurden zurückgewiesen, weil die Spitze Ihres aktuellen
Hinweis: Branches hinter seinem externen Gegenstück zurückgefallen ist. Wenn Sie
Hinweis: die externen Änderungen integrieren wollen, verwenden Sie 'git pull' bevor
Hinweis: Sie erneut push ausführen.
Hinweis: Siehe auch den Abschnitt 'Note about fast-forwards' in 'git push --help' für
Hinweis: weitere Informationen.

As recommended, I run git pull again (already expected that this is wrong), but now I got all messed up again.

I got conflicts with files outside of my pythonscripting changes

hhees@laptop-holger:~/workspace-openhab-build/github/openhab-addons> git pull
KONFLIKT (umbenennen/löschen): bundles/org.openhab.automation.pythonscripting/src/main/java/org/openhab/automation/pythonscripting/internal/wrapper/ModuleLocator.java zu bundles/org.openhab.binding.roborock/src/main/java/org/openhab/binding/roborock/internal/RoborockAccountConfiguration.java in HEAD umbenannt, aber in c003e5fed5337c261715fc517a263bf39311b2ae gelöscht.
KONFLIKT (ändern/löschen): bundles/org.openhab.binding.roborock/src/main/java/org/openhab/binding/roborock/internal/RoborockAccountConfiguration.java gelöscht in c003e5fed5337c261715fc517a263bf39311b2ae und geändert in HEAD. Stand HEAD von bundles/org.openhab.binding.roborock/src/main/java/org/openhab/binding/roborock/internal/RoborockAccountConfiguration.java wurde im Arbeitsbereich gelassen.
Automatischer Merge fehlgeschlagen; beheben Sie die Konflikte und committen Sie dann das Ergebnis.

The result I want is very simple. I just want to ignore the conflicts and overwrite them with my version. Thats all.

Nadahar · 2025-09-29T17:09:17Z

As recommended, I run git pull again (already expected that this is wrong), but now I got all messed up again.

I got conflicts with files outside of my pythonscripting changes

All was fine until here.... as I said, you almost never want to do git pull or git merge.

The reason you can't push is that you have rebased the existing commits so that they no longer "match" those already existing in the origin branch. A normal push only works if you just add new commits to those already there, it will be rejected when they mismatch (for good reason, this is to avoid overwriting/losing changes done either by yourself or others).

But, after rebasing, we must overwrite what's there, that's what we called a "forced push". To do that, simply run:

git push --force-with-lease

This will "force push" as long as the state of the remote still is as your local Git expects (that is, nobody else has pushed since you fetched). If you use just --force, this "safety" won't apply, so it will just overwrite even if somebody else has pushed something, so you should never use just --force.

If you do a git merge --abort now, you should once again be back to "the good place" where your push failed.

HolgerHees · 2025-09-29T17:19:09Z

All was fine until here

Hey, git push --force-with-lease was the secret. Thanks a lot. Now all conflicts are solved and pushed. :-)

Thanks a lot!

lsiepel · 2025-10-03T08:46:46Z

Please also adjust the signoff. Instructions are here:
https://github.com/openhab/openhab-addons/pull/19190/checks?check_run_id=51673492008

Signed-off-by: Holger Hees <holger.hees@gmail.com>

HolgerHees · 2025-10-03T11:00:38Z

Please also adjust the signoff. Instructions are here: https://github.com/openhab/openhab-addons/pull/19190/checks?check_run_id=51673492008

thanks for the hint. Is changed now.

lsiepel · 2025-10-03T12:06:56Z

I hope @wborn is able to look at this as my knowledge about both python and the scripting part of addons is limited to me.

florian-h05

Overall looks really good to me, I don't have much to comment.

I have reviewed everything except for all the console command stuff, I don't think this does much related to script engine and Graal stuff and is more "standard" Java code, hence I will leave the review to the add-on maintainers.

florian-h05 · 2025-10-05T15:10:17Z

...ipting/src/main/java/org/openhab/automation/pythonscripting/internal/PythonScriptEngine.java

+
+    @Override
+    public Object invokeFunction(String name, Object... objects) throws ScriptException, NoSuchMethodException {
+        if ("scriptUnloaded".equals(name)) {


Do I need to to the same for JS? Very likely yes?!

yes, it can happen that the engine is closed, before this is called.

Let me guess, you experienced this issue when using SCRIPT transformation?

In case your answer is yes, the following PR should fix the cause: openhab/openhab-core#5063.

yes, this was the scenario, I will change my code, when your fix is merged!

...in/java/org/openhab/automation/pythonscripting/internal/PythonScriptEngineConfiguration.java

...src/main/java/org/openhab/automation/pythonscripting/internal/PythonScriptEngineFactory.java

...tomation/pythonscripting/internal/scriptengine/InvocationInterceptingPythonScriptEngine.java

.../org/openhab/automation/pythonscripting/internal/scriptengine/graal/GraalPythonBindings.java

bundles/org.openhab.automation.pythonscripting/src/main/resources/OH-INF/config/config.xml

...src/main/java/org/openhab/automation/pythonscripting/internal/PythonScriptEngineFactory.java

Signed-off-by: Holger Hees <holger.hees@gmail.com>

florian-h05

LGTM now, thanks!
Again: I haven't reviewed the console command code.

Copilot

Pull Request Overview

Copilot reviewed 31 out of 41 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

bundles/org.openhab.automation.pythonscripting/src/main/resources/OH-INF/config/config.xml:1

Corrected capitalization of 'python' to 'Python' for consistency in UI text.

<?xml version="1.0" encoding="UTF-8"?>

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

.../openhab/automation/pythonscripting/internal/scriptengine/graal/GraalPythonScriptEngine.java

bundles/org.openhab.automation.pythonscripting/README.md

...ipting/src/main/java/org/openhab/automation/pythonscripting/internal/PythonScriptEngine.java

Signed-off-by: Holger Hees <holger.hees@gmail.com>

lsiepel

Thanks, LGTM

Could you add an upgrade notice to the distro repo for the user facing breaking changes?

HolgerHees · 2025-10-18T15:30:54Z

Thanks, LGTM

Could you add an upgrade notice to the distro repo for the user facing breaking changes?

@lsiepel just to let you know... Will add breaking changes soon. Currently I'm sick...

HolgerHees requested review from ccutrer, florian-h05 and wborn August 18, 2025 12:15

wborn requested a review from Copilot August 18, 2025 15:11

This comment was marked as outdated.

Sign in to view

wborn requested a review from Copilot August 21, 2025 17:33

This comment was marked as resolved.

Sign in to view

lsiepel reviewed Sep 3, 2025

View reviewed changes

...openhab.automation.pythonscripting/src/main/resources/OH-INF/i18n/pythonscripting.properties Show resolved Hide resolved

florian-h05 reviewed Sep 26, 2025

View reviewed changes

Nadahar reviewed Sep 27, 2025

View reviewed changes

HolgerHees force-pushed the next branch from c003e5f to 2bb7bda Compare September 29, 2025 17:17

HolgerHees mentioned this pull request Oct 1, 2025

[pythonscripting] README - typos #19413

Closed

HolgerHees added 6 commits October 3, 2025 12:59

initial changes

b72057a

Signed-off-by: Holger Hees <holger.hees@gmail.com>

implement review feedback

b95f817

Signed-off-by: Holger Hees <holger.hees@gmail.com>

improved README

dbdbf47

Signed-off-by: Holger Hees <holger.hees@gmail.com>

implement copilot feedback

0ffb022

Signed-off-by: Holger Hees <holger.hees@gmail.com>

generate i18n translations

9da9fcd

Signed-off-by: Holger Hees <holger.hees@gmail.com>

revert pythonscripting.properties

b53c5b7

Signed-off-by: Holger Hees <holger.hees@gmail.com>

HolgerHees added 3 commits October 3, 2025 12:59

revert pythonscripting_de.properties

4aad05d

Signed-off-by: Holger Hees <holger.hees@gmail.com>

adapt latest README feedback

3977538

Signed-off-by: Holger Hees <holger.hees@gmail.com>

fix readme

08b3158

Signed-off-by: Holger Hees <holger.hees@gmail.com>

HolgerHees force-pushed the next branch from 2990d70 to 08b3158 Compare October 3, 2025 10:59

florian-h05 reviewed Oct 5, 2025

View reviewed changes

...src/main/java/org/openhab/automation/pythonscripting/internal/PythonScriptEngineFactory.java Outdated Show resolved Hide resolved

implement latest Feedback

91107a3

Signed-off-by: Holger Hees <holger.hees@gmail.com>

florian-h05 approved these changes Oct 6, 2025

View reviewed changes

wborn requested a review from Copilot October 7, 2025 06:32

Copilot AI reviewed Oct 7, 2025

View reviewed changes

HolgerHees added 2 commits October 7, 2025 10:24

implement latest Feedback

5106738

Signed-off-by: Holger Hees <holger.hees@gmail.com>

Merge branch 'main' into next

0255b3a

Signed-off-by: Holger Hees <holger.hees@gmail.com>

HolgerHees mentioned this pull request Oct 11, 2025

[pythonscripting] typos “A(n) scope module”, plural form #19467

Merged

lsiepel approved these changes Oct 11, 2025

View reviewed changes

lsiepel merged commit cb38441 into openhab:main Oct 11, 2025
2 of 3 checks passed

lsiepel added this to the 5.1 milestone Oct 11, 2025

lsiepel added enhancement An enhancement or new feature for an existing add-on (potentially) not backward compatible labels Oct 11, 2025

Uh oh!

[pythonscripting] current marketplace version of pythonscripting next #19190

[pythonscripting] current marketplace version of pythonscripting next #19190

Uh oh!

Conversation

HolgerHees commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

openhab-bot commented Aug 19, 2025

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

florian-h05 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lsiepel commented Sep 27, 2025

Uh oh!

HolgerHees commented Sep 27, 2025

Uh oh!

Nadahar commented Sep 27, 2025

Uh oh!

HolgerHees commented Sep 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HolgerHees commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nadahar commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HolgerHees commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nadahar commented Sep 29, 2025

Uh oh!

HolgerHees commented Sep 29, 2025

Uh oh!

lsiepel commented Oct 3, 2025

Uh oh!

HolgerHees commented Oct 3, 2025

Uh oh!

lsiepel commented Oct 3, 2025

Uh oh!

florian-h05 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

HolgerHees commented Aug 18, 2025 •

edited

Loading

HolgerHees commented Sep 29, 2025 •

edited

Loading

Nadahar commented Sep 29, 2025 •

edited

Loading

HolgerHees commented Sep 29, 2025 •

edited

Loading

lsiepel left a comment •

edited

Loading