concept/rules: state that a rule cannot be executed in parallel #2573
Conversation
❌ Deploy Preview for openhab-docs-preview failed. Why did it fail? →Built without sensitive environment variables
|
|
This pull request has been mentioned on openHAB Community. There might be relevant details there: https://community.openhab.org/t/rules-and-concurrency/166577/33 |
|
If a rule cannot be executed in parallel, why does https://www.openhab.org/docs/configuration/rules-dsl.html#concurrency-guard say “If a rule triggers on UI events it may be necessary to guard against concurrency.”? |
That sentence is probably poorly worded. MainUI widgets can directly call a rule as its action. This usage is similar to running a rule manually (e.g. pressing the play button), running the rule directly through the REST API ( Note, I think that MainUI actually has some logic in place that will prevent you from manually running a rule that's already running on that rule's page. But that's logic built into the web page and it does not extend to UI widgets. |
|
I do not understand. Either OpenHAB-core, irrespective of web pages, ensures that a single Rule (or a script) can have a single running instance, or it does not make such promises. In this case, OpenHAB just waits for a rule to finish and if the rule is triggered in the meantime again, after it finishes, it is fired again. This is what the example above shows, this is how it works with transformations. If OpenHAB lets a rule under circumstances start running, before another instance of the same rule has finished, then this is completely different story and the conditions, when this can happen, should be described, so that locks guard against this. Do UI-widgets or REST API calls allow a rule (e.g. a rule containing |
There are many ways to run a rule. Some ways there are locks. Other ways there are no locks. Rules that are run based on a trigger are locked. Rules directly run (i.e. not run based on a trigger) there is no lock.
Right, the key word is "triggered" here. MainUI doesn't trigger the rule. It directly invokes it. That comment in the docs you point out is referring to this direct invocation.
Yes. You will have two of this rule running at the same time. Rule Action: Widget's Action: From a Script (JS in this case, note that Rules DSL lacks this ability) rules.runRule('RuleUID', { }, true);REST API: 
None of these ways to run a rule have locks. Only when a rule is triggered is it locked. |
dilyanpalauzov
force-pushed
the
rule_sequential
branch
from
9b18e40 to
725dcd3
Compare
|
I have changed the wording to state, that rules fired by their triggers cannot be executed in parallel, while rules executed by @Nadahar said at openhab/openhab-addons#19410 (comment), that each rule has a dedicated thread - . Does it mean, that when a rule is executed by
Is this logic in MainUI irrelevant (bypassed), if the same page is opened in two browsers? |
Actually, I believe that there's a "problem" when using |
I checked for timer triggers, in particular for logger = org.slf4j.LoggerFactory.getLogger("C")
scriptExtension.importPreset("RuleSupport")
global_i = 0
automationManager.addRule(
new org.openhab.core.automation.module.script.rulesupport.shared.simple.SimpleRule() {
@Override
Object execute(org.openhab.core.automation.Action module, Map<String, ?> inputs) {
Integer local_i = global_i++
logger.error("START " + local_i)
Thread.sleep(15000)
logger.error("END " + local_i)
}
{
triggers = [org.openhab.core.automation.util.TriggerBuilder.create().withId("a")
.withTypeUID("timer.GenericCronTrigger").withConfiguration(
new org.openhab.core.config.core.Configuration([cronExpression: "0/5 * * * * *"])).build()
]
}
})prints This trigger fires every five seconds, but because of By their nature the other timer triggers |
|
@dilyanpalauzov I didn't mean timer triggers, but timers used in scripts. I'm not entirely sure how they are used to execute rules, I've just read it referenced. However, you should keep an eye on this, it might change how this works: openhab/openhab-core#5069 |
Are profiles also restricted to run sequentially, in the same way as transformations are?Sequential executions for profiles is stated at https://github.com/dalgwen/openhab-addons/blob/java223/main/bundles/org.openhab.automation.java223/README.md#concurrency with these words: Timers in Rules and ScriptsMy understanding is that a timer in a script is in essence a new thread, and then locks can be justified, if the timer-body references variables, which exist outside the timer, and and the rule is retriggered - the rule accesses the same variables at the same time - once after being retriggered, and once from the timer. That said, I think timers and locks in rules and scripts do not need extra text. I have such a rule (DSL), which changes a value of a String item (e3_colour) to contain a colour (red, orange, yellow). The longer the device is on, the hotter the colour. |
|
If a profile cannot run in parallel to itself, does this apply only for profiles written in automation language? (There is such statement for transformations written in automation language). Or for all profiles? |
concepts/rules.md
Outdated
|
|
||
| Rules can be executed, either when fired by their triggers, or when called explicitly - from a script, another rule, MainUI widget. | ||
| When a rule is fired by its trigger, the execution of one and the same rule does not happen in parallel. | ||
| If a rule is triggered for execution, while the rule is currently run, it will be queued and run later. |
There was a problem hiding this comment.
If a rule is triggered for execution, while the rule is currently run, it will be queued and run later.
... while a rule is currently running*...
configuration/rules-dsl.md
Outdated
| ### Concurrency Guard | ||
|
|
||
| If a rule triggers on UI events it may be necessary to guard against concurrency. | ||
| If a rule is executed not fired by its triggers, but explicitly - from a script, from another rule, from a MainUI widget - the rule can be started, before its previous execution has ended. It may be necessary to guard against concurrency. |
There was a problem hiding this comment.
Each sentence should be on a separate line.
The wording of the first sentence is awkward. I recommend something more like:
"If a rule is explicitly run from another script, rule, or MainUI widget instead of a trigger the rule can be started before the current execution has ended."
Yes, unless openhab/openhab-core#5069 is merged, in which case it isn't anymore. That said, I agree that using a lock like you do above isn't hard, but there are some complicating factors. First, according to @rkoshak there are bugs with Rules DSL which means that In addition, there are other scripting languages than Rules DSL where doing something like this is much more cumbersome. And, many people don't really understand the concept at all, why there is a need for locks or how you use them. For reasons not clear to me, concurrency seem to be one of those things that only some people "get". There are so many concurrency mistakes made, also by developers that are otherwise very capable. |
dilyanpalauzov
force-pushed
the
rule_sequential
branch
from
725dcd3 to
ad42664
Compare
On the text rewording hereI have taken the suggested text over. In:
I have removed “or” before MainUI, as the list is not exclusive: HTTP POST Bugs in Rules DSL …
Where are these bugs described? Can I have an example reproducing this? Can a lock be handled as a try-with-resources, so after (Moving Rules DSL out of openhab-core is another solution to the problem, or it lets isolate the problem either in DSL or in all rule executions systems - right now I do not know where it is.) Can non-script transformations run in parallel?Sometimes - yes, sometimes - not! Not only transformations from automations scripts are executed sequentially (one transformation per thread), but all transformations. I modified the MAP transformation to sleep for 5s with the patch to delay transformation.map with 5s--- a/bundles/org.openhab.transform.map/src/main/java/org/openhab/transform/map/internal/MapTransformationService.java
+++ b/bundles/org.openhab.transform.map/src/main/java/org/openhab/transform/map/internal/MapTransformationService.java
@@ -84,8 +84,13 @@ public class MapTransformationService
@Override
public @Nullable String transform(String function, String source) throws TransformationException {
+ logger.error("A/" + Long.toString(Thread.currentThread().getId()) + "/ "
+ + java.time.ZonedDateTime.now().toString().substring(11, 22) + " input=[" + source + "]");
+ try {
+ Thread.sleep(5000);
+ } catch (Exception e) {
+ }
Properties properties = null;
-
Matcher matcher = INLINE_MAP_CONFIG_PATTERN.matcher(function);
if (matcher.matches()) {
properties = cachedInlineMap.computeIfAbsent(function, f -> {
@@ -120,15 +125,22 @@ public class MapTransformationService
if (target == null) {
target = properties.getProperty("");
if (target == null) {
+ logger.error("B/" + Long.toString(Thread.currentThread().getId()) + "/ "
+ + java.time.ZonedDateTime.now().toString().substring(11, 22) + " input=[" + source
+ + "] new Exception");
throw new TransformationException("Target value not found in map for '" + source + "'");
} else if (SOURCE_VALUE.equals(target)) {
target = source;
}
}
-
+ logger.error("C/" + Long.toString(Thread.currentThread().getId()) + "/ "
+ + java.time.ZonedDateTime.now().toString().substring(11, 22) + " input=[" + source + "] output="
+ + target);
logger.debug("Transformation resulted in '{}'", target);
return target;
}
+ logger.error("D/" + Long.toString(Thread.currentThread().getId()) + "/ "
+ + java.time.ZonedDateTime.now().toString().substring(11, 22) + " input=[" + source + "] new Exception");
throw new TransformationException("Could not find configuration '" + function + "' or failed to parse it.");
}
I have the MAP transformations called fromand the seven r9, r9a...r9f items appear in r9.sitemaps: For the results below I visit However I do not understand the results. These are the results from the MAP transformation logging in /var/log/openhab/openhab.log (for brevity I removed the date at the start of the lines and ` [ERROR] `)That transformation is executed in threads 316, 331, 485, 785. Thread 316 executes the transformation 7 times, the other execute the transformation 8 times. Indeed So, one and the same transformation - script or not - is executed several times in the same thread, when
How can it be generalized when one and the same transformation runs several times (sequentially) in the same thread, and when in parallel using different threads? Parallelism in Profiles from transformationsI see no point in looking at how this works, before it is clarified when is one and the same transformation executed several times in the same thread and when in a different thread. (As I cannot know if the findings for profiles will be because of how profiles work, or because of how transformations work.) |
I'm not sure that they are described except for in forum posts, probably. @rkoshak might have an idea how to trigger it. A From what I understand, this is a pretty fundamental flaw with xtext/xtend (I'm not sure exactly where) that nobody has been able to solve, and it's one of the reasons why focus was shifted away from Rules DSL. If you can't trust a
As I'm sure you know, I'm very much against moving it out of core, since it's the one BIG benefit of Rules DSL as I see it - that it's NOT an add-on. But, understand one thing: The different script languages have different challenges, some more than others. No one person have the overview of all this, and what comes up as examples are somewhat random. If you want to follow this to the end, and really find all the problems, I guess you will have to actually try to do it in all the different languages, and see what barriers present themselves. So, the focus on "if the bug is in Rules DSL or not" isn't really that important here. Rules were never meant to or designed to run in parallel with themselves as I understand it, so there are probably lots of small decisions that would have to be different, if they were meant to be. What I don't understand is why you'd want a rule to run in parallel with itself. It sounds like a mess even without concurrency issues. A rule normally does some evaluation of something and then makes a change. If this evaluation → change process goes on in parallel, how can this be anything but a mess? One instance of the rule is evaluating while the other is applying the action, the next iteration has decided something different partly based on what the first one did, and so it will immediately change the decision made. It sounds like a recipe for "random behavior" to me. When it comes to parallel execution of transformations, I'm not familiar enough with the code to know how it's done. But, what you're seeing could mean that they, in theory, run in parallel, but that for other reasons (the code that launches the transformations), they often aren't in effect run in parallel. There are lots of moving parts here, and one lock (in the "wrong place") can often be enough to prevent things from happening in parallel. |
I haven't used Rules DSL in many years and do not care to. But it used to be the case that if you had a type error in the try block (e.g. tried to do an operation with two Objects of incompatible types), Rules DSL would throw and catch and log a NullException. Because it caught its own exception, the rule just exits and is never given the chance to run the finally. There were other errors that got generated deep in the Xtend call stack which cause the rule to just stop. The exception never bubbles back up to the rule so the catch and finally never execute.
I pushed really hard for this. But it's been decided. It's not going to happen. So we work with what we have. Ultimately, .items files, .things files, .sitemap, et. al. legacy file formats are defined by and parsed using Xtext/Xtend too. Moving Rules DSL to become an add-on doesn't actually remove much of anything from core because just about everything Rules DSL needs is also needed by these other file formats. But I'm also not 100% certain something like this isn't also possible in other rules languages. There's a whole stack of Java stuff that a rule's scripts are running on top of. If there is an exception anywhere in that stack and the exception doesn't flow up to the script, you can never guarantee that the finally block will be run in any rule script. In practice, it probably doesn't happen that way, certainly not often, but there is also no guarantee.
I suspect it's how the profile is called that makes the difference, same as with rules. The sitemap needs to make some REST API call to apply the transformation in the label so, as when a rule is run directly like that, it probably bypasses any locks. That's my guess anyway.
This is exactly how it worked in OH 1.x and 2.x (for those not using the Experimental rule engine). Imagine the fun we had to deal with when using an "Item state updated" trigger on a Group with an aggregation function. The rule got triggered N-1 times where N is the number of members of the Group in rapid succession. I spent roughly 75% of my time on the forum helping users find, diagnose, and solve errors caused by the same rule running more than once at a time. It was indeed a mess. As I see it, there are only two ways you can make it so the same rule can run more than once at a time, and this applies to all the languages.
1 isn't feasible because it eliminates pretty much every use case people have for rules. 2 isn't feasible because, particularly on SBCs like RPis, the amount of time it takes to parse and instantiate the script is long. For a poorly created rule in Rules DSL (i.e. forcing types where you don't absolutely need to) it can take dozens of seconds. We can't afford that kind of latency every time the rule runs. I'm certain we could find ways to amortize and reuse stuff so it's not quite that bad, but it's never going to be down to acceptable levels of latency. (Note, it used to be the case not that long ago that rules were not loaded and parsed/compiled until the first time the rule was triggered, and the user experience was horrible). |
Let me explain you my understanding of the difference between Xtext and Xbase/Xtend. XText defines the grammar for sitemaps, items, things, rules files. In grammar for rules I mean "start possibly with imports, then Xbase and Xtend are only in charge for the magic block above and DSL transformations. Xbase and Xtend parse the code (only the magic block) and then can either interpret it, or generate .java files. I think in openHAB it is interpreted, as it seems to run even on systems without javac, without implementation of In addition Xtext provides some services for Language Server support, and Xbase/Xtend might also provide such services. In another addition Xtend/Xbase is used to compile parts of openHAB, e.g. bundles/org.openhab.core.model.sitemap/src/org/openhab/core/model/sitemap/SitemapRuntimeModule.xtend, but this is only compile time dependency. So sitemaps depend on Xtend for their compilation, but at run time (after openHAB is shipped) model.sitemap, model.persistance, model.thing do not need Xtend, they need only Xtext (for parsing files). All Xtend files in openHAB coreThat said, removing Rules DSL and DSL transformation from openhab-core means also removing Xtend and Xbase. It does not mean removig Xtext. In fact Xtend/Xtext would be very fine, if it can run without the Rules grammar. That is to be able to write .xtend files in automation/jsr223/, which are either interpreted or converted to .java files, but do not have this Execting a rule to itself in parallel
To run a rule in parallel would actually mean that the only thing a rule does is to start a new thread and execute a function in that thread. Then the rule returns (immediately), before the function in the thread has completed. This can be done even now and as it can be done now, the rules can run with the same amount of ScriptEngines as now and share data in the same way as now. The only drawback is that a rule, which returns immediately cannot return a value. But this value is useful only, if a … Okay, above we talk about rules, but we mean rule actions. So if a rule has many actions, when each action returns, it passes data to the next action. And when the action returns immediately, then the next action cannot use the return value of the previous action. Except the previous action returned a Promise… and who needs several actions to a rule… That said, as it is now, as people can fire threads from (slow) rules and return immeditely, the same rule can run virtually in parallel to itself. The other only drawback is that starting new threads/executing actions in thread pools will eat more resources than now. However the above does not apply to transformations, as they need a String return value immediately. |
|
This pull request has been mentioned on openHAB Community. There might be relevant details there: https://community.openhab.org/t/rules-and-concurrency/166577/62 |
I don't understand what you're trying to say here. If you launch new threads (which you can basically do anywhere where you're allowed to "run code", those new threads aren't "rules". They are just threads, they don't interact with OH as part of a rule, they aren't treated as such, and you're "completely on your own" as to what these achieve. I don't think such "abusive techniques" need to be a part of the regular documentation. If doing that serves a purpose for you, please go ahead and do it, but for me, it's very difficult to understand what I could achieve by doing something like this. |
|
As for removing Rules DSL from OH, you can go tilt at that windmill. As far as I'm concerned, the developers/maintainers have spoken definitively. Rules DSL is not going to be removed from core.
And those separate runs of the function share data. In fact, they don't just share data, they share memory.
What gets logged in step 5? What gets logged in step 6? In both cases This is why you can't run the same script more than once at the same time except under the restrictions I outlined above. And yes, this is exactly the sorts of things we had to deal with back when rules could run at the same time.
Anyone using "simple UI rules" (i.e. using any action that isn't an "Inline Script"). Granted this is exceptionally limited right now but Scripts are not the only Rule Action. |
|
The focus of the current changeset is to write down what cannot run in parallel, what can run in parallel, what only sometimes runs in parallel - file based rules, UI rules, file-script, UI scripts, actions, profiles, profiles linked to scripting transformations, profiles linked to non-scripting transformations, scripting transformations, non-scripting transformations… Currently it is not clear when a non-scripting transformation reuses a thread (runs sequentially - is started, after it finished) and when it runs in a new thread (parallel to itself). But as an exception to what I just said:
why does openhab/openhab-core#3233 exist as it is? |
Because he was also in favor of moving Rules DSL to an add-on. But the consensus of the rest of the maintainers was to leave it part of core. So that Issue was never worked and it probably should have been closed. |
dilyanpalauzov
force-pushed
the
rule_sequential
branch
from
ad42664 to
aace84a
Compare
4 participants
I create a file rules/a.rules:
and then change
rsix times fast. openhab.log contains:So both rules are executed six times, but SLOW Rule waits its own execution to end, before starting from the start.