Too much LUA?

[Mark F]

I'm becoming increasingly worried about the amount of stuff that needs to execute in the LUA environment. What I mean is there are things that are being done in LUA scripts that take a LONG time to complete and during this time no other threads can use the LUA environment.

On my current development system, this has the effect of delaying all event processing as, for example, a LUA based UI is initialized. This means plugin and LUA script developers need to take into account that they may not be able to execute LUA code for a MINUTE or more. In this non-deterministic environment, "time-out" logic becomes painful or, sometimes, worthless.

I have toyed with the idea of creating and managing multiple separate LUA environments inside the serial plugin to get around this problem; but, before taking this radical step, I was wondering if anyone else sees "too much LUA" as a problem?

It is quite possible that, instead, I am just worried about nothing. :)

[Rob H]

You have a good point - it would be good to be able to run a LUA script as a separate thread. e.g. LDJ's Rescan can take quite a while to run.

[Promixis]

I certainly see your concern.... especially with the serial plugin has time critical pieces. There is a multithreaded non standard distro of lua 5 which could solve this problem. Or you could have a seperate enviroment on a differnet thread with all the issues attached with that. However, I haven't seen problems with Girder becoming bogged down running too much Lua. The HAI Omni and LCD Master scripts are probably doing a 1000 lines of lua a second with 3-5% cpu use on a P 1.5. The real problem I see is a lua chuck that takes a long time to run and blocks the rest of Girder ie. we now have a lua zip library and that certainly can hog the system for many seconds....

[Rob H]

A way of letting a Lua script yield control back to Girder would help. Although there are reentrancy issues obviously.

[Mark F]

I agree, Mike. LUA runs perfectly fast. Some of the operations we are asking it to do block for a long time. That is the problem.

[Mark F]

Actually, Rob, LUA 5 contains support for cooperative multi-threading. (It is called coroutines in the documentation)

In the past, cooperative multi-threading, in an open development environment, hasn't worked out very well. If you are old enough ;), remember Windows 3.x and how a single application could freeze the UI while it did something? Same problem. One poorly written script spoils it for everyone.

[quixote]

I never realized that this was an issue before, but now that you have pointed it out I can see ho this could be a serious problem. I think that there are going to be an increasing number of people that use Girder for more than just their HTPCs. I for one am planning on managing a security system that uses motion sensors, door magnets, and cameras. I also will be managing my door entry system. I can't think of anything presently that may interfere with the setup, but as a possible example for the future, what would happen if someone was trying to enter the front door and I was browsing the LUA XMLTV Guide? Or if there is a complicated automatic procedure happening while there is a break-in and as a result there are no video clips taken and emailed to me at work in time before the computer is disconnected?

Maybe I'm just paranoid and thinking of a worst-case scenario. :oops:

[Mark F]

Perhaps the solution, then, is to have a design guide that states that any interface that you expose in LUA should NOT block, unless that is it's only purpose (see win.Sleep()). If the operation will take a "long time", there should be a notification interface (callback, event, etc.) to announce completion.

Our new problems will then be:

How does this get enforced?
How long is a "long time"?
How do we get the LUA script development community to use an event driven or finite state machine paradigm instead of the brute force straight line method? (see #1)


Probably not a good idea after all. :(

[Mark F]

quixote - That is exactly the type of stuff that worries me. :( Mission critical, 24/7/365.

[Ron]

Mike and I spoke briefly and I agree, this is a problem waiting to happen and I have be thinking of a solution. One thing that Mike suggested is this:

http://www.cs.princeton.edu/~diego/professional/luathread/home.html

I am experimenting with this right now.

[Rob H]

Actually, Rob, LUA 5 contains support for cooperative multi-threading. (It is called coroutines in the documentation)


I'd forgotten about those - not having done much Lua 5 yet. But will that actually yield to Girder?



In the past, cooperative multi-threading, in an open development environment, hasn't worked out very well. If you are old enough ;), remember Windows 3.x and how a single application could freeze the UI while it did something? Same problem. One poorly written script spoils it for everyone.

Oh, I'm plenty old enough :)

[Promixis]

quixote - That is exactly the type of stuff that worries me. :( Mission critical, 24/7/365.

Girder must be able in this state. I do not know if we can control what plugin authors do in that level of detail other then for users to be aware that they may have issues when plugins/code blocks Girder...

However, any script/plugin can cause problems even in a multithreaded enviroment. Our goal is to provide the most versaitile and robust enviroment for doing event driven automation.

[Rob H]

luathread sounds promising

[Mark F]

Mike - Yup. Bad code is bad code. Hard to fix that from the outside. :( Isolating bad stuff and minimizing it's impact is the key. I don't have a complete solution either.

I like what I've read about luathread. I like it a lot.

OT: Thanks once again for being so responsive to problems, ideas and suggestions. I basically started this thread by saying, "Your baby is ugly". Instead of arguing you started looking for cosmetic surgeons. It isn't said enough: You guys ROCK!

[Ron]

Thanks Mark, that is the way Girder development has always worked and I hope we can keep it that way!

ps. That luathread thing might just work!

[Ron]

Okay consider this an alpha of an alpha :-)

Examples here:
http://www.cs.princeton.edu/~diego/professional/luathread/examples.html

I did make some modifications to the library but none of them are visible to the end user except that Girder doesn't crash on a syntax error :-)

[Ron]

Sorry guys missing file in the archive. Here is the complete archive :-)

[Promixis]

Here is an example using the Lua Zip Extensions library.....

Quite cool to watch.


--[[

Create a ZIP file using the files in the Girder directory

--]]



function ZipCreateandAdd (zf,files)
local zipobject = zip.CreateZip (zf,zip.CREATEZIP)

if not zipobject then -- failed to created the zip
print ("Could not create ",zf," error",err)
return
end

local drive,dir

drive,dir = win.SplitPath (files)

for fa in win.Files (files) do

if math.band (fa.FileAttributes, win.FILE_ATTRIBUTE_DIRECTORY) == 0 then
local res,err = zipobject:Add (fa.FileName,drive..dir..fa.FileName)
if not res then
print ("Failed to add ",fa.FileName," error ",err)
else
print ("Added ",fa.FileName)
end
end
end

zipobject:Close ()
end

local gpath = win.GetDirectory ("GIRDERDIR").."\\"
local zf = gpath .. "test.zip"

thread.newthread (ZipCreateandAdd, {zf,gpath.."*.*"})

for x =1,100 do
print (x)
win.Sleep (100)
end

[Ron]

Note the for loop at the end is for cosmetic purposes only :wink: The thread will go on even if the lua code finishes.

[Mark F]

Am I being pre-mature? Can I start re-designing the serial plugin to take this into account? :D

[Ron]

Hang on a little try testing it though, I am adding some stuff to this library to make it more usefull. Like returning thread handles and waitforthread functionality. Also I need to keep track of running threads so I don't pull girder away from beneath a running lua thread.

[Ron]

Okay Mark GO!

http://www.promixis.com/girderplugins/girder-4-a8-MT.exe

This includes updated documentation so you know what is available.

[Mark F]

Cool! 8)

FYI, in the thread example in the help, the less than character (<) is messed up. It has been replaced with &tl on my Win98 machine.

EDIT: Same thing on my WinXP Pro machine at work.

[Mark F]

Houston, I have a problem here. ;)

I've started utilizing threads in the serial plugin. I block a thread and wait for a serial event to occur before releasing it back into the LUA environment. The problem is that when I try to close Girder, the serial plugin doesn't release the thread and I get a message that Girder couldn't kill the thread. I'm watching for the GIRINFO_SCRIPT_BEFORE_STOPPED message and that is not occuring.

Any suggestions?

[Ron]

Housting, this is Phoenix Mission control, we have you covered.


Any suggestions?

I'll look at what is going on.

[Ron]

Mark give this one a try.

[Mark F]

:) Much better. Thanks.