rules_overview.md: elaborate about concurrency support in rule engines#2565
rules_overview.md: elaborate about concurrency support in rule engines#2565dilyanpalauzov wants to merge 1 commit intoopenhab:mainfrom
Conversation
✅ Thanks for your pull request to the openHAB documentation! The result can be previewed at the URL below (this comment and the preview will be updated if you add more commits).Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
This pull request has been mentioned on openHAB Community. There might be relevant details there: |
|
I asked at https://community.openhab.org/t/java223-scripting-script-with-java-on-openhab-5-0-1-0-6-0-0-0/159853/45: For clarification, if |
|
I'll also answer here. For all rules no matter the language being used:
tl;dr: only one instance of any given rule can run at a time. It is impossible for a single rule to be running two or more times in parallel. All this locking is within a single rule. You can have as many separate rules running at the same time in parallel as your machine is able to handle.
Some rules languages have further restrictions. For example, GraalVM JS has a limitation that disallows any context from being accessed by more than one thread. Each rule represents a separate context. When a context is accessed by more than one thread, a MultiThreadedException when the second parallel access is attempted. To prevent these exceptions, the GraalVM JS add-on implements additional locking on the timers. The rule/script and the timer created by that rule/script share a context so you cannot have a timer running at the same time as the rule that created it. Therefore, GraalVM JS adds locks so only the rule or only one of its timers can be running at any given time. There is no locking on the shared cache nor on rules that are run directly instead of being triggered. If someone puts anything more than a primitive in the shared cache and two separate GraalVM JS Scripting rules attempt to access that Object at the same time, a MultiThreaded exception will occur (a warning is logged when you attempt to put something more than a primitive in the shared cache in GraalVM JS). When a rule is called directly from MainUI or from another rule, there is no locking so MultiThreaded exceptions will occur if two or more attempt to call the same rule at the same time. Best practices:
|
It's not a single thread, it will be different threads, but the locks make sure that just one of them are allowed inside the locked section at a time. So, there is no concurrency, anything that utilize e.g. JavaScript will have to "wait in line" until other threads are done using the JavaScript engine. JavaScript doesn't support multithreading as a language, so this isn't really any different from all the JavaScript running in your browser or other places, even though the "web workers" is an attempt to work around this limitation for browsers. Python supports multiple threads but only allow one of them to run at any time, so it's also without concurrency. This isn't a limitation of JSR223, but of these languages themselves, so there's really nothing that can be done to make them handle multithreading. If scripts don't "sleep" or do blocking calls to e.g network resources/web pages/APIs, it won't really be that much of a problem, since the scripts will run quickly and thus release the locks quickly. But, if scripts do these things... there will be queues piling up. |
I think it's important to separate whether the language can run multithreaded and whether the way GraalVM works with these languages allow access to "stuff" created by the languages to be accessed by multiple threads at the same time. It's the latter that's a limitation for GraalVM JS as we use it (there actually is a way to run GraalVM JS with Java that does allow multuithreaded access to the context, but it's not really something that we can make deployable in an OH context. It requires installing and running a separate process (IIRC) instead of embedding it as an OSGI service. This all isn't strictly a case where "the language can't do it". It's more of a case of "the way we need to use GraalVM in OH doesn't allow it.
Yes, but if you write a rule that blocks like that and triggers often enough to matter, it will only block that one rule, not all rules. At least that's the way it's supposed to work. I haven't tested it since OH 3 though. Note this is a huge improvement over the way it used to work in older verison of OH where one rogue rule could kill all your rules waiting for access to a thread to run in. |
The limitations of GraalVM, and how OH uses it, the restrictions in OH itself, all matter. The reason I've been focusing on these languages lacking multithreading support is that, regardless of all the other limitations, if they didn't exist, you still couldn't let them work with multiple threads at once because the languages lack the mechanisms to do it. They simply don't have syntax/techniques to handle concurrency. So, how could it ever work, regardless of other imposed limitations? JS and Python seem to have picked a slightly different strategy to avoid having to deal with concurrency, but the result isn't very different. In the world of JS, threads doesn't really exist, and instead it has evolved into a quite complicated asynchronous system via the "event loop", that will call different things from a single thread in sequence. As a consequence, you never really know when something will run, just that it probably will "at some time in the future". Python allow threads, but to avoid having to deal with concurrency, they have the "Global Interpreter Lock" (GIL). It's a lock/mutex/synchronization that only allows one thread access to the Python interpreter (which is the entity that makes anything happen, much like the "script engine" for JSR223). So, even though you can have threads, they will spend their time waiting for their turn, only one can run at any given time. This means that shared objects can safely be accessed by different threads, since there is no actual concurrency, so the language doesn't have to deal with ways to handle concurrency (like locks/mutexes). So, despite their approach being different, the end goal seems to be the same: Avoid having to deal with concurrency, with the price being that they can't really utilize multiple cores. But, both being more "light scripting languages" than full programming languages, probably considered this as an acceptable tradeoff. The end result remains that you lack ways to handle concurrency in the languages themselves. The "web workers" in browsers tries to add threads to JS, but not by integrating it into the language, but my making JS code use an API that basically implements what is lacking in the language itself.
That's not true, as far as I can understand, for JavaScript and Python with these engine-wide locks. One "rogue rule" in one of these languages can "kill"/pause all rules using the same language. I don't have the full overview, but it might be possible to make it work differently by having one In many ways, this is a quite hard "design question" for how all this should ideally work, what solution are possible, desirable, and what the price of the different decisions are. I certainly don't have all the answers. |
|
Interestingly, GraalVM claims to support concurrency if some rules are followed - so I'm questioning if the "engine lock" is really necessary or desirable: https://www.graalvm.org/22.0/reference-manual/js/Multithreading/ Exactly what are the circumstances that result in this java.lang.IllegalStateException: Multi threaded access requested by thread.. |
|
https://www.graalvm.org/22.0/reference-manual/js/Multithreading/ says the limitations of multi-threading for JavaScript:
This happened here openhab/openhab-addons#13022 - A rule starts a timer (
openhab/openhab-addons#13022 says “Multi threaded access requested by thread …” |
But in an OH rules and transformation's context, you have access to all of Java too which can. For example, if I want to sleep for some reason in a JS rule I can use: Java.type('java.lang.Thread').sleep(5000); We are not in a pure environment. It's a hybrid between JS and Java (or Rules DSL and Java, or Groovy and Java, ...). I'd have to experiment to find the proper syntax, but you should be able to create a thread that way too. And all the locking classes in Java are available too.
It's easy enough to test. Given the following rules: configuration: {}
triggers:
- id: "1"
configuration:
cronExpression: "* * * * * ? *"
type: timer.GenericCronTrigger
conditions: []
actions:
- inputs: {}
id: "2"
configuration:
type: application/javascript
script: console.info('Rule is running!');
type: script.ScriptActionconsole.info('about to sleep');
Java.type('java.lang.Thread').sleep(3000);
console.info('done sleeping');I should never see the log "Rule is running!" in between "about to sleep" and "done sleeping". And yet... Even if I change the sleep to a busy wait we can see that two JS scripting scripts are running at the same time. I'm not sure why it's not sleeping for the full 30 seconds though but that's not relevant to this discussion. console.info('about to sleep');
//Java.type('java.lang.Thread').sleep(3000);
let end = time.toZDT('PT30S');
let i = 0;
while(time.toZDT().isBefore(end)) {
i++;
}
console.info('done sleeping');These would not be possible if only one JS rule could run at a time.
I can't speak to the cost, but that is how it appears to work. Something is in place that allows JS rules to run in parallel.
Indeed. I said so above. But we can't deploy GraalVM JS in a way that we can take advantage of that without adding a bunch manual steps beyond installing the add-on. Those steps are not compatible with OSGI.
I don't know. It might be that the JS Scripting add-on wraps it. I do know in openhab.log we see an exception that starts with "MultiThreaded". It may be MultiThreadedAccessException or just MultiThreadedException. You ccan generate it if you construct a test where a rule gets called (not triggered, called directly, in JS that would be Each rule has its own thread. If they all ran in the same thread, there would be no chance for a multithreaded exception. |
Because 3000 milliseconds, as in Thread.sleep() are 3 seconds. The above example shows, that two JavaScript rules can be executed at the same time. My explanation is that there are two distinct |
|
But getting back to this PR. I don't think it's correct that JS and Python do not have concurrency. Separate rules do run in parallel (see the test above for proof). All rules languages including GraalVM JS and Python have access to all of Java including ReentrantLock, Thread, etc. So there is access to concurrency primitives. However, GraalVM JS and maybe Python have a concurrency restriction in that nothing from any one rule (script really) can be accessed from more than one thread at the same time. But that does not mean that these scripting languages do not support concurrency at all. Therefore, the edits of the PR are incorrect.
I can, but it shouldn't matter. A rule is a rule once it's created. The only difference is how how the rule gets created in the first place: i.e. loaded from JSONDB or created through code using the APIs. It will take me a bit to do that though. It might be tomorrow but I'll try later. I thought I had typed 30000. Doh!
This needs to be inside a rule too, right? Otherwise we are not testing the same thing,. If it's not in a rule we are testing whether multiple .js files can be loaded at the same time, not whether multiple rules can run at the same time. Also, it needs to be in automation/js. GraalVM doesn't use jsr223. |
I mean in all cases everything should be in a single .js file, so that one ScriptEngine handles it. I though you start with console.info, outside rule, then install the rule from the same.js code, then sleep and console.info. But you can try also with two rules, as long as they are loaded from a single .js file. |
That won't show anything. It certainly isn't a duplicate of the test above. The test above is two rules, one a full rule and the other just a Script (since it gets run manually). There's no code tab on Scripts in MainUI so just pasted in the code. But it is a rule.
I seem to be having a problem with my script recognizing the automatic imports. This might take longer. I might be hitting a bug. I have to fight that before I can get some results. One thing that might cause confusion is that when using .js files to create rules, you are actually writing a JS program whose job is to create rule(s) when it's run. OH loads the file and runs the script and obviously it needs a ScriptEngine to execute that rule creation code. But once the rules are created, they are independent from that ScriptEngine and it appears get their own ScriptEngine. So if I put a sleep outside of a rule in a .js file, all I'll accomplish is adding a delay between the creation of the rules in that file depending on where I put the sleep. |
Yes, the automatic imports are not working with automation/js files with OH 5.1.0 Build 4836, even if for me “Auto injection for all scripts, including file-based scripts and transformations“ is selected. This works const { rules, triggers } = require("openhab")
console.info("START")
rules.JSRule({
name: 'ABC',
description: 'ABC',
triggers: [triggers.GenericCronTrigger("* * * * * ? *")],
execute: (data) => console.info('Rule is running!'))
})
Java.type('java.lang.Thread').sleep(30000)
console.info("END")Result: As long as there is |
|
OK, I managed to work around the problem. Indeed there appears to be a bug with automatic injection of the variables for file based rules. I'll file an issue for that later. But color me surprised. The blocking rule did block the cron triggered rule. Code: const { rules } = require('openhab');
rules.when().cron('* * * * * ? *')
.then( (event) => {
console.info('Rule is running!');
})
.build('Every Minute', 'Logs every minute for testing', [], 'everyMinute');
rules.when().item('TestSwitch').receivedCommand()
.then( (event) => {
console.info('About to block');
Java.type('java.lang.Thread').sleep(15000);
console.info('Done blocking');
})
.build('Blocking', 'Sleeps for 15 seconds', [], 'blocking');Logs: I don't like this difference in behavior at all. I tried moving the blocking rule to its own file. When the rules are defined in separate files they do not block each other. I think I understand why, and I should not have been surprised. All rules defined in a given .js file share the same context. They don't magically get a new context once they are created. At a minimum the Script Actions for all the rules need to be passed to the rules that's created even if everything else gets created as Java in the rules engine (which I don't think is the case). Because they share the same context, none of these script actions can run at the same time. So to go back to your original scenario:
And by "running" I mean executing script conditions and script actions. They can still be triggered but the triggers will be queued and work on in sequence. |
|
Ah, bug after bug. With the script below, or something nearly identical to it, when I change ABC to ABC2 and save, I get: and when I repeated the above several times, I think I hit a deadlock : nothing happens. So stopping and starting openhab, putting under /etc/openhab/automation/a.js: const { rules, triggers } = require("openhab")
console.info("START")
rules.JSRule({
name: 'ABC',
description: 'ABC',
triggers: [triggers.GenericCronTrigger("* * * * * ? *")],
execute: (data) => {
console.info('ABC START')
Java.type('java.lang.Thread').sleep(3000)
console.info('ABC END')
}
})
rules.JSRule({
name: 'DEF',
description: 'DEF',
triggers: [triggers.GenericCronTrigger("* * * * * ? *")],
execute: (data) => {
console.info('DEF START')
Java.type('java.lang.Thread').sleep(3000)
console.info('DEF END')
}
})
Java.type('java.lang.Thread').sleep(30000)
console.info("END")output is
I come to the same conclusion with the example above. Okay, we found it out! So to my question:
Yes, splitting in many files does increase the parallelism! I will change the current PR. |
True, but I'm not sure if you can get access to the intrinsic locks from Java, they aren't objects, but are rather a part of every object. Maybe it would be possible, but I don't know how to do that from Java either, you use the Maybe there are ways around this, but I can't imagine this whole thing would be very user-friendly or at all viable for most users.
We're talking about different things. Whenever I talk about a rule, I don't mean a
That means that the What I'm talking about is two normal rules. Try to create two of them that use JS and let one sleep, and I'm pretty sure you'll see a different result. These will, as far as I can understand, all share the
I'm not sure that's true. I'm talking about within the same script engine instance. It says that as long as the context isn't shared between threads, multiple threads can actually run scripts in the same script engine. However, it also says that "JavaScript objects" cannot be accessed by multiple threads, but this is where I don't follow quite... because, if the context isn't shared, how would two threads get access to the same "JavaScript object"? So, to me it looks like you probably could run things in parallel of you kept the contexts separate. However, I'm not sure how all the stuff the helper libs insert works. Are there "JavaScript objects" in there, are there any form of global variables available across script engines?
Aren't you saying that you must do something that breaks this isolation, like using timers or invoking a different rule, to trigger such an exception? In addition there are things like
IF they use different script engine instances, which is only the case for
This is all about the different script engine instances, as I already tried to point out. The locks follow the script engine instance, so regardless of which thread is trying to execute, only one will be allowed to run within the same script engine instance. I'm having more and more doubt if this is the correct solution the more I learn about it.
How did you reach that conclusion? This isn't the case, as far as I can understand, for script engines that implement the lock. They will all have to wait for each other, because they share the script engine instance.
This depends on how the add-on creates script engine instances, but seem to be the case for the JS add-on at the moment.
Yes, because there is one script engine instance per file. The reason for this restriction is the
What immediately intrigues me here is: Why is the thread interrupted? A thread that is interrupted (has been interrupted in the past, so that the interrupted flag is set), will immediately throw
I would actually hold off documenting this until things are straightened out. How much of this has changed recently? Are the script engine locks new? If so, is it even certain that this solution is viable - I assume that it's supposed to prevent the "multithreaded exception", but I fear that it will cause much more problems than it solves. Or, has it been this way for some time, and something else changed in 5.1? |
|
To throw another wrench into the machinery, there is also this: It's supposed to protect the |
|
Regarding the mentioned pythonscripting. Pythonscripting can fully run asynchron in the way it means for Python (GIL) You can create Threads and Timers inside a script file and they fully run in parallel. The lock from PythonScriptEngine is only used during initial eval calls. This are internals and does not affect the runtime context of python itself. The scenario where you have a sleep inside the execute function has nothing to do with python threading. It is just a symptom of how openhab works.
This is why you should never to complex things inside the execute statement. If you have something complex todo, just create a Thread and do it there. |
|
Python is a programming language. When implemented in Jython it does not have Global Interpreter Lock. The CPython implementation has Global Interpreter Lock and the way to achieve concurrency is using multi-processing (many processes instead of many threads). CPython 3.13 can be compiled without Global Interpreter lock. Some more tests: JavaScript: Each rule is executed in its own threadconst { rules, triggers } = require("openhab")
console.info("START " + Java.type('java.lang.Thread').currentThread().getId())
rules.JSRule({
name: 'ABC',
triggers: [triggers.GenericCronTrigger("* * * * * ? *")],
execute: (data) => {
console.info('ABC START ' + Java.type('java.lang.Thread').currentThread().getId())
Java.type('java.lang.Thread').sleep(3000)
console.info('ABC END')
}
})
rules.JSRule({
name: 'DEF',
triggers: [triggers.GenericCronTrigger("* * * * * ? *")],
execute: (data) => {
console.info('DEF START ' + Java.type('java.lang.Thread').currentThread().getId())
Java.type('java.lang.Thread').sleep(3000)
console.info('DEF END')
}
})
Java.type('java.lang.Thread').sleep(30000)
console.info("END")The above shows that each rule is executed in its own thread, even if only one of the threads, created by a single Parallelism of one and the same TransformationHow can I check, if the same Groovy/Java223/JavaScript/Python transformation can run in parallel? I have created three logging transformations:
public class D {
public Object main() {
String s = java.time.ZonedDateTime.now().toString().substring(11, 19);
org.slf4j.LoggerFactory.getLogger("J").warn(s);
try {
java.lang.Thread.sleep(3000);
} catch (InterruptedException e) {}
return "A " + s + " - " + java.time.ZonedDateTime.now().toString().substring(11, 19);
}
}
String s = java.time.ZonedDateTime.now().toString().substring(11, 19)
org.slf4j.LoggerFactory.getLogger("L").warn(s)
java.lang.Thread.sleep(3000)
"G " + s + " - " + java.time.ZonedDateTime.now().toString().substring(11, 19)
s = (new Date).toISOString().substr(11)
console.log(s)
Java.type('java.lang.Thread').sleep(3000)
"J " + s + " - " + (new Date).toISOString().substr(11)Whatever I try I cannot get the same transformation logging in less than 3 seconds gaps (time at the beginnig of the lines in openhab.log). Parallelism in RulesI have changed the two rules, which trigger on cron event and are blocking each other, with item change trigger: const { rules, triggers } = require("openhab")
console.info("START " + Java.type('java.lang.Thread').currentThread().getId())
rules.JSRule({
name: 'ABC',
triggers: [triggers.ItemStateChangeTrigger('r9_power')],
execute: (data) => {
console.info('ABC START ' + Java.type('java.lang.Thread').currentThread().getId())
Java.type('java.lang.Thread').sleep(3000)
console.info('ABC END')
}
})
rules.JSRule({
name: 'DEF',
triggers: [triggers.ItemStateChangeTrigger('r9_power')],
execute: (data) => {
console.info('DEF START ' + Java.type('java.lang.Thread').currentThread().getId())
Java.type('java.lang.Thread').sleep(3000)
console.info('DEF END')
}
})
Java.type('java.lang.Thread').sleep(30000)
console.info("END")This does not print two START or two END in a row, so that each rule blocks the other rule. The equivalent in Groovy: scriptExtension.importPreset("RuleSupport")
@groovy.transform.Field logger = org.slf4j.LoggerFactory.getLogger("logger2")
automationManager.addRule(new org.openhab.core.automation.module.script.rulesupport.shared.simple.SimpleRule() {
String name = "GHI"
List<org.openhab.core.automation.Trigger> triggers = [
org.openhab.core.automation.util.TriggerBuilder.create().withId("t").withTypeUID("core.ItemStateChangeTrigger")
.withConfiguration(new org.openhab.core.config.core.Configuration([itemName: "r"])).build()
]
@Override public Object execute(org.openhab.core.automation.Action module, Map<String, ?> inputs) {
logger.warn("GHI BEGIN")
Thread.sleep(3000)
logger.warn("GHI END")
return null
}
})
automationManager.addRule(new org.openhab.core.automation.module.script.rulesupport.shared.simple.SimpleRule() {
String name = "JKL"
List<org.openhab.core.automation.Trigger> triggers = [
org.openhab.core.automation.util.TriggerBuilder.create().withId("t").withTypeUID("core.ItemStateChangeTrigger")
.withConfiguration(new org.openhab.core.config.core.Configuration([itemName: "r"])).build()
]
@Override public Object execute(org.openhab.core.automation.Action module, Map<String, ?> inputs) {
logger.warn("JKL BEGIN")
Thread.sleep(3000)
logger.warn("JKL END")
return null
}
})does execute the rules in parallel:
I guess this eval is done after the input is compiled and there is no equivalent of org.openhab.automation.jsscripting.internal.threading.ThreadsafeWrappingScriptedAutomationManagerDelegate.java for Python. If the above example, presented in JavaScript and Groovy, is rewritten in Python(GraalVM), how does the code look like and are the two rules executed in parallel? |
|
This is whats happen in python scripting So it looks like each CronTrigger ist triggered in its own thread. And it works asynchron. and instead of using Java classes, below is a pure python example of your code, which is doing exactly the same. |
|
Another example using Item Events |
|
So, I can only speak for pythonscripting, but please remove the part "Has no concurrency" from pythonscripting. |
|
I am convinced, that Python can run things in parallel, but I do not understand what implications has
|
|
This is very difficult to discuss, as so many different terms are used. Asynchronous and concurrent are entirely different things. Asynchronous only means that things don't run in a sequential order. Concurrent means in parallel, which means "happening at the same time". Neither Python nor JS can do concurrency in themselves. But, obviously, if you run multiple script engines (essentially isolated JS or Python execution environments), these environments can work in parallel - they aren't even aware of each other. But, that's like starting a single threaded program 5 times and claim that it's multi-threaded, because it runs multiple times at once. |
|
openhab/openhab-addons#19410 is about to add in jsscripting/README:
Is this the same for all automation/python/*py files? I agree this at this moment is hard to discuss for many reasons. My suggestion is to propose documentation changes and then these changes can be discussed. |
|
For normal endusers of pythonscripting, they can just ignore the internal engine lock. |
I think the purpose is to describe when things block each other. |
If you narrowly only focus on that, sure. To me, getting the overview is more important, because that's what you need to see if there can be done improvements to "the architecture" itself, where locks are needed and why. To do that, separating the different concepts/concerns is important. The old "don't plan, just slap on some locks until it works" approach doesn't really give good results. It usually ends with too much locking, but not necessarily everywhere it's needed, and very high contention. Instead of solving the design problems themselves, workarounds are then often found for some isolated issue, that makes things worse in other areas, and you end up with something that never really works properly or effectively, that is tweaked "constantly" for new, surprising behavior. That's why I'm in favor of planning the strategy, and then implement that. It will usually give the best result, and the least amount of surprises. But, much of this has been done already, so it's not really up to me to "design" anything here, still, this is where my mind goes when discussing problems: what is the overall strategy to handle this, is the strategy flawed, and if so how? Is the strategy actually followed? Are there things being done that don't conform to the strategy? |
In OH the shared cache is one way.
I think yes.
Yes, a timer created by the same script would and could trigger the exception without the locks in the add-on preventing that.
No, it's implemented by core. JS Scripting will print a warning to the log if you attempt to put something more than a primitive into the shared cache though.
Remember this is a PR to update the docs. The docs represent how OH works right now, not how it could be fixed maybe someday in the future. The forum or a PR on the add-on is a much better place for that. Right now we have definitive answers as to the behaviors. All managed rules can run in parallel. All rules defined in separate .js files can run in parallel. All rules defined in the same .js file run in sequence. The end users have no control over whether they are creating a SimpleRule or a "normal" rule. So all that discussion is irrelevant to this PR.
See my experiment above showing that two managed JS rules running at the same time, Also see @florian-h05's explanation on the forum thread.
I don't think anyone is saying that isn't important. But this is a documentation PR. It's not appropriate here because this repo isn't where anything will change if we decide something needs to change. And even if we decide nothing needs to change, we still need to document how it works now. Most of the discussion above was relevant in it helped us understand how OH actually works now so we can make appropriate documentation changes. But we can't do anything about whether it could be done better or not in this repo. |
I guess the thread is interrupted, because a rule, loaded from a
It is in practice on hold off. I do not know what has changed, it all started with a discussion of multithreading in the Java223 community add-on: many people have different opinion how multi-threading is done in openHAB, so writing some clarifications would be useful. There is no relation to changes in 5.1.
As demonstrated by other examples, there is no Cron processing thread, that was locked, two cron rules can be executed in parallel, provided they are not created from the same |
I realize that we're probably ending up quite far out on a branch here, but the whole reason I've been talking about intrinsic locks is that if you use an object that already has locking implemented, you must use the same lock as is implemented. Since this is most often the intrinsic lock, you can't actually (as far as I can tell) acquire that lock. But, I realize when thinking about it that this probably isn't that relevant, it gets "too advanced". If you only want to protect some object, you could absolutely use a Java
Is the OH shared cache something that is implemented in OH, across scripting languages? It could probably be made thread-safe (if it isn't already) in the Java code then, so that whenever you retrieve something from the cache, proper locks are applied and a copy is made. That object could then be manipulated locally without locks, but the cache wouldn't automatically be updated as a consequence. For that to happen, you would have to "store" the object in the cache again. If this is how this already works? That said, I don't know how "JavaScript objects" can be stored in Java, but I guess they are serialized to JSON or something like that?
I would think that global variables would live in the context object, and not in the libs. But, again, I could be wrong. These things are essential to understand what must and must not be protected against concurrent access.
The reason I'm asking is that I'm thinking that if the only ways you can get into this "problem" is via timers or calls to other rules, maybe something could be done to avoid that in core instead. I would imagine, without knowing the details, that it should be possible to make a timer also run in the "rule thread" instead of a new one. Calling other rules might be slightly more complicated if this must be blocking/retrieve a result, but if not (if the script just triggers another rule and then move on), it could probably be solved in a similar way to timers.
So you have any idea what it's called internally, to improve my chance to actually find and look at how this is done?
I'm aware that a lot of this discussion goes beyond the PR. The PR is also closed now, so it's not that big of a deal. But, it would probably be better if the discussion was in a place where it would be easier to find later or for others.
I'm not sure about managed rules, did you ever test with two managed rules? It depends on how the script engine instance is handled.
I agree that the end user probably have no idea about that, but if you're about to make assertive statements, and the reality is different for the different types of rules, how do you do that without asserting something that is wrong?
I never saw that you made two managed rules, is this what you mean?
Because, frankly, you only showed one of the rules, and just the "action part" of the other one - which is why I assumed that the second rule was a But, you're right that if it's like @florian-h05 says that a new script engine instance is created for every execution (of "normal" rules), they will be able to run independently (but it sounds like a very expensive/slow way to do it).
Fair enough. Parts of me screams "leave this one alone", it will be too much frustration to try to get to the bottom of or make improvements to. But, if I'm unable to listen to myself, where do you think this should be discussed? On the forum, or in a "dedicated issue"?
Even if the rule was changed and the current execution was attempted terminated, the execution of the new version shouldn't execute from a thread in an interrupted state. But, perhaps I misunderstood what was happening. I'm still somewhat curious where the interrupt came from though.
Sadly I can't, because GraalVM won't work in my dev environment, and there has been zero interest in figuring out why this doesn't work. So, to run the script I would have to set up a "regular OH instance", but that would leave me without all my dev tools, so it wouldn't let me examine what's actually going on.
👍 All the more important to get what is stated absolutely correct. |
I have not used it, but I think it is implemented in OH-core. Converting it to thread-safe thing ... will introduce more locks, and more mutexes means less parallelism.
I do not know more, why the thread was interrupted. The only thing I could do is describe how to reproduce this interrupted sleep. |
No. The whole point of making the cache thread-safe was so that you could drop the script engine lock, which would increase parallelism. And, I wasn't thinking of a cache that "locked" objects, but one that duplicated them on retrieval - which should only hold a lock for a very short time. But, depending on how the cache is implemented, this might require major changes in how it's used and thus be nonviable. |
|
I'll add that to see what threads does what, it's useful to add the <Console name="STDOUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%-5.5p] (%-20.20thread) [%-36.36c] - %m%n"/>
</Console>The above will print the 20 last characters of the thread name as a part of each log line. |
Yes. In fact, you can access Objects in the cache from different languages. If I put something in it at key "foo" using JS, I could pull it out in a Python script or Rules DSL. It's only JS (as far as I'm aware) that has a restriction that the contexts need to be locked. And any changes made to the cache will impact all the automation add-ons.
I created an issue to implement exactly that. Maintainers decided that it was a misuse of the shared cache to put Objects in it like that so they implemented the warning instead.
But if they are written with the assumption that there will be one instance per ScriptEngine that already happens. so there would be no reason to do anything special.
I do not, but as I said this idea has already been rejected. Based on how it's used I suspect it's backed by a Map of some sort. But I think it's a Registry (like ItemRegistry).
Yes of course. That was the first test I did. And I have already said that I did at least twice. Once again, I tested with two managed rules, both rules in the same .js file , and each rule in it's own .js file. And @florian-h05 explicitly stated how the script engine is handled for managed rules. Each script action and each script condition gets its own ScriptEngine.
Because I tested every combination of ways to create a rule available right now in JS. I can assert without reservation how JS rules work right now as a result. When and if a new way to create rules in JS is implemented, these docs can and should be revisited. In the mean time, we have all the information we need from a JS perspective at least, to understand the current behavior.
I explained that above as well. The second one was a MainUI -> Settings -> Script and there is no "Code" tab for Scripts defined that way. But a Script in that context is just a managed rule that consists of a single Script Action, no triggers and no conditions and is tagged with "Script". MainUI presents these two differently. It would be nice if a Script did have a code tab. 🤷♂️
Either place. Probably the forum first because assuming some changes are to be made, it's not clear whether they would be done in core or the automation add-on(s) or both. |
In my memories it was OH JavaScripting, which invented the cache mechanism, and later this mechanism was implemented in openhab-core and removed from https://github.com/openhab/openhab-js + org.openhab.automation.jsscripting. You can look in the history of the openhab-js and in particular openhab/openhab-addons@438552d485357d9060862, to see how the cache was implemented in JavaScript and, if the cache is implemented in JavaScript, you should have been able to put inside the old implementation JavaScript objects. (I have never used it.) How the cache works for all rule (script) engines is described currently at https://www.openhab.org/docs/configuration/jsr223.html#cache-preset. |
|
JavaScript objects can be passed to Java, GraalJS takes care of conversion. Though I don‘t think there are many complex types that end up in the shared cache, as language interoperability only really works for primitive types. |
|
This pull request has been mentioned on openHAB Community. There might be relevant details there: https://community.openhab.org/t/rules-and-concurrency/166577/2 |
|
This pull request has been mentioned on openHAB Community. There might be relevant details there: https://community.openhab.org/t/rules-and-concurrency/166577/33 |
6 participants
Based on discussion at https://community.openhab.org/t/java223-scripting-script-with-java-on-openhab-5-0-1-0-6-0-0-0/159853/27, and looking for
git grep [^a-zA-Z]Lock bundles/org.openhab.automation.*, I see that onlyorg.openhab.automation.pythonscripting.internal.PythonScriptEngineandorg.openhab.automation.jsscripting.internal.OpenhabGraalJSScriptEngineimplementLock. My understanding is that in these cases there is a single thread, which executes all the Python/GraalVM code, and another thread for executing all the JavaScript/GraalVM code. Hence the last sentence of the changeset.