PDA

View Full Version : Implementing comet long polling...



shaun5
January 24th, 2012, 01:58 PM
I have a web interface to Girder that mainly runs on my iPad. I get state updates every second via ajax requests for a json string. The frequency of requests has a few drawbacks, so I'm going to convert to long polling. Anyone else doing (or tried) this? Anyone currently using Lua coroutines for anything? What are the connection limitations for the built in webserver?

Ron
January 24th, 2012, 02:33 PM
It's been quite a while since I touched the webserver code but a quick scan over the code base shows me that each response has it's own lua state, so potentially long polling might work. Simply add a 'wait' in the .lhtml page. Let me know what you find or run in to.

shaun5
January 24th, 2012, 03:07 PM
I'll add a wait on a test page just to see if there are issues there, but will ultimately need something different. I want the lowest lantency with the least amount of network traffic.
Without coroutines, I don't see a way to recall and respond to all the pending requests, so...

My idea is to: check a version or timestamp of the current states and send state updates (how the ajax requests work now without the check) OR if states are current, add a coroutine function to a table, pause the coroutine. Once something affects a state restart all coroutines in the table and send the updated states. Does this seem like the best way or is there a better way to access the open connections??

shaun5
January 24th, 2012, 10:32 PM
socket.sleep(10) will delay the response without tying up the CPU.

I thought maybe putting it inside a coroutine:

myco = coroutine.create(function ()
socket.sleep(10)
end)
coroutine.resume(myco)

Would allow me to wake up the thread by yielding the coroutine so that I may immediately send the response, but when I add a script action (simulating new data):

coroutine.yield(myco)

I get an error: attempt to yield across metamethod/C-call boundary

Ron
January 25th, 2012, 07:39 AM
you need to use thread.newmutex / thread.newcond to do this. Are you familiar with these?

The manual has a short example on how to use this. I don't think coroutines will work for this.

shaun5
January 25th, 2012, 10:20 AM
I forgot to report on my last response that using socket.sleep(10) keeps the ajax request open and a response after the delay is received by the client, so long polling is possible with Girder!

Ron: I am not familiar with thread.newmutex / thread.newcond. The link in the manual to 'LuaThread Manual' is dead...

Ron
January 25th, 2012, 10:22 AM
you'll find an example inside the Girder manual. Go to Lua Libraries and find the luathread section.

shaun5
February 5th, 2012, 07:14 PM
Ron: I changed my approach to implement this, but I think either solution is going to have the same issue that needs resolving. Attached is my cometreqjson.lhtml file (named .txt because the .lhtml aren't uploadable). It is used the same way as the previous ajaxreqjson.lthml (built on your ajaxreq.lhtml). My cometreqjson.lhtml will wait for a change in the requested variables to send a response that only includes new data. If no change is made within 10 seconds, it just sends a json string that indicates no change.

The client side is changed to allow a longer response. I have improved the client side with the use of try / catch to account for server responses that are not JSON encoded (this eliminates client side errors that stop the functions from chaining).

I am also adding classes while a control is active on the client side to ignore server responses for only that portion of the client interface. This eliminates the need to start and stop the requests from the server that cause the client to be updated with non current data.

Below is an excerpt showing the revised javascript (I wish the forum code button showed up in Safari):

jsonOBJ = {};
function checkGIRDER() {
$.ajax({
type: "GET",
url: "cometreqjson.lhtml",
data: {ID : browserID, Data : "transport.devices.AIR.AIR_volume,transport.devices .TIVOHD.TIVOHDstatus"},
async: true,
cache: false,
timeout:50000,
success: function(data){
checkDATA(data);
setTimeout('checkGIRDER()',100);
},
error: function(XMLHttpRequest, textStatus, errorThrown){
$("#text1").text("Comet Timeout ERROR: " + textStatus + " (" + errorThrown + ")");
setTimeout('checkGIRDER()',100);
},
});
}
function checkDATA(data) {
$("#text1").text(data);
try {
jsonOBJ = jQuery.parseJSON(data);
for (var key in jsonOBJ) {
if (jsonOBJ.hasOwnProperty(key)) {
if (key == 'transport.devices.TIVOHD.TIVOHDstatus') { $("#TIVOHDstatus").text(jsonOBJ[key]); }
else if (key == 'transport.devices.AIR.AIR_volume') {
if (!$("#controlREADOUT").hasClass('activeREADOUT')) {
$("#controlREADOUT").text(jsonOBJ[key].toFixed(1));
jsonOBJ[key] = parseFloat(jsonOBJ[key]);
$("#progressbar").progressbar({value: jsonOBJ[key]});
}
}
}
}
}
catch(err) { $("#text1").append(" INCORRECTLY FORMED JSON DATA...");}
}
setTimeout('checkGIRDER()',100); //not sure if setTimeout starts a separate thread in the client browser, but I am using it just in case...

THE PROBLEM: Girder is delaying the processing of the responses from my TCP based transports (haven't tested RS232 responses). Is there only one TCP socket thread? How can this be fixed??

shaun5
February 9th, 2012, 01:38 PM
Ron, what do you think?

Ron
February 9th, 2012, 01:53 PM
I looked at this two days ago when you posted it but could not find a reason for the delay on the transport devices when you hold a lua state on the webserver. The webserver and the transport devices have their own threads and lua states, I'm puzzled.

I will have to do testing to see what is going on.

Hmm I noticed you are using thread.sleep. a quick google gives this poor fella (unresolved) http://cboard.cprogramming.com/c-programming/138623-2-threads-1-socket-sleep-causing-problem.html

Can you use win.Sleep instead?

shaun5
February 9th, 2012, 07:25 PM
After more testing, the problem originates at the client (this may have been the problem the whole time...). Mobile Safari is delaying sending additional ajax requests while waiting for the response.

Ron
February 9th, 2012, 07:34 PM
that's too bad. Can you install a different browser?

shaun5
February 9th, 2012, 08:25 PM
More testing and I have finally isolated the problem...

The good news is my code is good on both ends (client and server).
The bad news is I was wrong (again). Girder delays requests from clients (single client sending additional requests or a second client sending a request) while the cometreqjson.lhtml file is executing. (This should be something easy to replicate and see) So... I need Girder to accept additional browser requests while others remain waiting. Any solutions??

shaun5
February 14th, 2012, 01:03 PM
Ron, do you see any solutions?

I was also looking into websockets, but may be faced with the same type problem (Girder blocking while the port is in use by a single client).

Ron
February 14th, 2012, 01:50 PM
After looking at that code again. I'm afraid there is no solution based upon HTTP communications. The server is implemented using asynchronous IO. As such it uses only one thread. We're now trying to place synchronous IO on top of it (the sleep). Which then sadly blocks the whole server.

Alternatives:
Flash / Java based object that creates a raw tcp socket. Then using the transport classes create something that sends an even if the object of interest changes. Then reload the data over HTTP as normal.

shaun5
February 15th, 2012, 04:44 PM
Ron, do you see a way to use Apache as the Girder webserver? This would provide a way to implement new technologies (like websockets) without having to start from scratch...

shaun5
February 19th, 2012, 10:22 AM
Ron, I need the lua module crypto OR the module md5 for the md5 handshake authentication of websockets. How can I add it?

Ron
February 19th, 2012, 04:37 PM
No problem. I've added md5 to bitlib for lack of a better spot and to get it done quick for you.

Here is the usage. If I move it at any point the usage will remain the same with the exception of the first line (bit.newMD5)


require('bit')

local md5 = bit.newMD5()

md5:update("The quick brown fox ")
md5:update("jumps over the lazy dog.")

print(md5:final())

gives E4D909C290D0FB1CA068FFADDF22CBD0

which is correct according to wikipedia.

shaun5
February 19th, 2012, 08:03 PM
Thanks Ron, that got me working! Still have some cleanup to do, but the basic framework is now functional. I can send data to ALL the connected clients simultaneously or individually from Girder. Clients can also respond individually to Girder through the open channel.

This is really going to change the way Girder and webpages interact.

Ron
February 20th, 2012, 07:17 AM
That is fantastic!!! Can't wait to see what you have created!! Awesome work.

shaun5
February 20th, 2012, 01:55 PM
Are the methods included the md5 library you provided to compute the 'string hashed by SHA1' and then 'base64 encoded'? If so, what are they (I can't find the API anywhere)? This would allow me to get protocol 6 working which covers Chrome, IE9+, and Firefox. Safari and Mobile Safari use protocol 0 and that is what I have working...

What is going to be the best way to package this? Right now, I just have three scripting actions in a folder in my GML: server (fires on fileloaded and scriptenable),broadcaster (server started),closer(fires on fileclose). I was thinking one file with all the code and just have script actions to call functions. Or is there a better way?

Ron
February 21st, 2012, 07:02 AM
md5 doesn't do sha1/base64. I'll see if I can put together something cleaner for more permanent use.

shaun5
February 21st, 2012, 10:56 PM
Ron, I need to store a json string or two per client and have gotten stuck. How do I reference the p1 object name within the p1:callback function? I had envisioned something like WSclients[p1]['data1'] = cp1, but not sure if that will work if I had the object name... Attached is a .zip with websocket.GML, client.html, and the referenced jQuery file so you can see the basic functionality in Safari. You'll have to change the IP within the client.html file to your Girder IP. 'Onload' websocket.GML setups up the websocket server using port 8080. The 'basic send' script action will broadcast a message to all the connected clients.

Anyone wanting to give this a try will need the bit.dll file Ron posted on page 2 dropped into the Girder5 directory...

Ron
February 22nd, 2012, 07:32 AM
function dehex(str)
str = string.gsub(str,'[^%a-zA-Z0-9]','')
return (string.gsub(str, '..', function (cc) return string.char(tonumber(cc, 16)) end))
end
function fixkey(str)
local num = ""
string.gsub(str,'%d', function(d) num = num .. d end)
local _,n = string.gsub(str, ' ', ' ')
local i = tonumber(num)/n
return string.char(math.mod(i/256^3,256), math.mod(i/256^2,256), math.mod(i/256,256),math.mod(i,256))
end
function websocketresponse(key1, key2, end8)
require('bit')
local cat = fixkey(key1) .. fixkey(key2) .. end8
local md5 = bit.newMD5()
md5:update(cat)
return dehex(md5:final())
end

function WScallback(p1,p2)
if ( p2 == transport.constants.event.CONNECTIONCLOSED ) then
print("WEBSOCKET Connection Closed by Girder")
return
end
if ( p2 == transport.constants.event.NEWCONNECTION ) then
print("WEBSOCKET New Connection")
WSclients[p1]=true
--actually need two different terminators \r\n for header and string.char(255) once the websocket is open...
p1:Callback(transport.constants.parser.TERMINATED, string.char(255), 2000, function (cp1,cp2)
if ( cp2 == transport.constants.event.CONNECTIONCLOSED ) then
print("WEBSOCKET Connection Closed by client (can be the response to Girder sending closing message)")
p1:Close()
WSclients[p1]=nil
elseif (cp2 == transport.constants.event.INCOMPLETERESPONSETIMEOU T) then
local connecting = string.Split(string.gsub(cp1,'\r',''),"\n")
local response = 'HTTP/1.1 101 WebSocket Protocol Handshake\r\n'
response = response .. 'Upgrade: WebSocket\r\n'
response = response .. 'Connection: Upgrade\r\n'
--response = response .. 'Sec-WebSocket-Protocol: chat\r\n'
for i in connecting do
if(string.sub(connecting[i],1,6) == 'Host: ') then
response = response .. 'Sec-WebSocket-Location: ws://'..string.sub(connecting[i],7)..'/\r\n'
elseif(string.sub(connecting[i],1,8) == 'Origin: ') then
response = response .. 'Sec-WebSocket-'..connecting[i]..'\r\n'
elseif (string.find(connecting[i],'Key1') ~= nil) then
websocket_key1 = string.sub(connecting[i],string.find(connecting[i],':',1)+2)
elseif (string.find(connecting[i],'Key2') ~= nil) then
websocket_key2 = string.sub(connecting[i],string.find(connecting[i],':',1)+2)
elseif (string.len(connecting[i]) == 8) then
ending8 = connecting[i]
end
end
p1:Write(response)
p1:Write('\r\n')
p1:Write(websocketresponse(websocket_key1, websocket_key2, ending8))
print('WEBSOCKET Client Connected.')
elseif (cp2 == transport.constants.event.RXCHAR) then
--need to save a client's string
print('WEBSOCKET Client Sent: ' .. string.sub(cp1,2))
end
end)
return
end
end
if ( WSclients ) then
for c,v in pairs(WSclients) do
c:Close()
end
end
if ( WSt ) then
WSt:Close()
win.Sleep(1000)
end
WSclients = {}
WSt = transport.New(transport.constants.transport.GIPLIS TEN)
if not WSt:Open(nil, 8080) then
print("Could not create WEBSOCKET Server.")
WSt:Close();
WSt = nil;
end
WSt:Callback(transport.constants.parser.TERMINATED , '\r\n', 2000, WScallback)
print("Starting WEBSOCKET Server...")
--gir.TriggerEvent("WS_broadcast", "100", 0)


In this code? Looks like you are already doing that reference?

shaun5
February 22nd, 2012, 10:27 AM
Your right. I think I tried to nest additional tables without first defining WSclients[p1]={}. I'll rewrite it and let you know.

Did you try the demo?

Ron
February 22nd, 2012, 04:25 PM
I'm sorry I didn't try yet. It's crazy how little time there is in a day.

shaun5
February 23rd, 2012, 07:46 AM
No problem! The correction worked. I now have an additional function (working in its own thread) that loops every 250ms to send differential json data updates (just like the cometreqjson) to open clients. The data monitored is defined individually by each client. Clients without websockets degrade to Ajax automatically.

If the same data is requested to be monitored by multiple clients, Girder still copies and monitors the data per client (so this could be improved if you had 4+ clients - it would just require a common data pool that is updated as clients open and close connections). The solution presented is probably best case for up to 3 or 4 simultaneous clients. Any more and the overhead of the pool will become negligible compared to the individual data copy comparisons.

I have a client side update cache to write then I will present the complete solution. The update cache will allow the client to ignore ignore updates while the client fiddles with the interface. This is nessessary to accomodate latency between everything involved in the chain (client, network, Girder, equipment)

shaun5
February 28th, 2012, 06:17 PM
Everything is done (excluding the additional browser support - still need the Lua library) and running. I'm still on the fence on adding a common data pool (I'll add it with if there is any interest). If someone is interested in using websockets, let me know and I'll upload everything and write it up.

Ron
February 28th, 2012, 06:39 PM
That's very cool. Which functionality do you need again? Just SHA-1 and Base64?

shaun5
February 28th, 2012, 06:43 PM
Ron, that is correct.

Ron
February 29th, 2012, 01:59 PM
Alright added SHA-1 and Base64 encode and decoding. Also I've added "init" calls to prevent the need for continually creating new objects by allowing hash object reuse.


require('bit')

local md5 = bit.newMD5()
md5:init()
md5:update("The quick brown fox ")
md5:update("jumps over the lazy dog.")

print(md5:final())

local sha1 = bit.newSHA1()

sha1:init()
sha1:update("The quick brown fox jumps over the lazy dog")
print(sha1:final())

local str = "The quick brown fox jumps over the lazy dog"

local es= bit.base64encode(str)

print(es)

local ds = bit.base64decode(es)

print(ds)

shaun5
February 29th, 2012, 09:37 PM
Ron: Thanks for the bit.dll. Using the WebSocket API, I know I can correctly generate the protocol 6 response. I've got Chrome and Firefox upgrading the connection from connecting to open, but I can't send or receive a response yet. Safari and Mobile Safari still work nicely...

shaun5
March 1st, 2012, 08:45 AM
Ron: Poor assumptions, lead me to wrong conclusions... I have been receiving data from Protocol 6 all along.

Both protocol 0 and 6 start with HTTP headers terminated with \r\n. Once protocol 0 is 'open', the data is formatted in both directions as: HEX 00 .. data string .. HEX FF. Protocol 6 data is different. Is just using STREAM vs. a PARSER going to be the best option? Or is there a way to start with the \r\n parser and once the protocol is determined for the client the parser is changed according to the requirements for each client??

Adding support for both protocols is going to take some work! A derivative of protocol 6 is probably the future, so it is probably a worthwhile improvement...

Ron
March 1st, 2012, 08:56 AM
I think stream will be your best bet in this case.

shaun5
March 7th, 2012, 02:07 PM
Update: Everything working... Ron, is there a way to set up the webserver to forward a port 80 request with a ws. prefix to my websocket transport listening on port 8080?

shaun5
March 10th, 2012, 07:53 PM
Update (if anyone is interested): I've added event triggering. This eliminates ajax_sendevent.lhtml for my websocket clients. Not sure if there is anything to be gained, but it seemed odd to not use the open websocket connection for two way communication.

Mike C
March 11th, 2012, 01:18 PM
hi shaun, do you have any screen shots and code to post?

mike

shaun5
March 11th, 2012, 09:21 PM
Mike: I can't think of any relevant screen shots, just tell me what you are looking for and I'll help. Did you try my posted example? Do you have a working webpage for Girder? If so, can you post it?

Except for adding the files, there is zero configuration in Girder. Unfortunately, I don't think the Gider collective has standardized on how the javascript should be written, so there is some adaptation (or conversion to my methods) to be done there. I was waiting to write it up in hope Ron discovers or shows us (me) a way to forward the websocket.

Ron
March 12th, 2012, 07:25 AM
you could open a http request with curl inside a webpage and forward it to the websocket...

shaun5
March 12th, 2012, 09:36 AM
Ron: Everything I know regarding port redirection is server specific (IIS or Apache). I didn't think the Girder webserver is either, what is it?

Ron
March 12th, 2012, 09:43 AM
Oh I see,... yeah that won't work. While you can do HTTP requests on behalf of a web page (using some lua curl inside a lhtml page) then you're stuck as that page cannot do the websockets protocol.

shaun5
March 12th, 2012, 12:00 PM
What is the webserver built on?

Ron
March 12th, 2012, 12:05 PM
in house code no external web server code was used.

shaun5
March 28th, 2012, 06:00 PM
Is anyone using this or even tried it?

sirbooker
March 29th, 2012, 07:17 PM
yes i did try it briefly but was unable to get anything to happen
can you do a little write up as what is to happen and where to place all the files
thanks paul.

shaun5
March 30th, 2012, 12:44 PM
Paul: You should just be able to add the .GML file, drop the bit.dll file in the correct directory (I'm away from Girder, but think it is just the girder5 directory), drop the cometreqjson.lhtml file in the http directory, adjust/add the javascript to your webpage and be up and running. I failed to add a note in the javascript to change the IP to the IP of your girder machine. If your using my old ajaxreqjson.lhtml, it shouldn't be much a change for you javascript. If you are having problems, just post your webpage and I'll take a look..

pfeifer
May 14th, 2012, 06:54 AM
Hi shaun5,
I am playing with your nice job.
But I would appreciate if you could post a full example (webpage + config file) just for retreive the value of the two following examples
1) One for a simple variable e.g. volume
2) Another one for a table variable e.g. Lights["on","on","off","on","off"]

My goal is convert my polling based project to your project based.

Thanks a lot
Sandro

shaun5
May 17th, 2012, 10:07 PM
I didn't setup a dummy data structure to test, but I think this should work. You have to change the IP in the code to go to your server and have the files earlier in this thread...

shaun5
September 21st, 2012, 08:16 PM
IF (and probably a BIG IF), anyone is using my websocket. Replacing the 'else' code within the WS.write function with the following code will update to RFC-6455 (for iOS 6.0 and Safari 5.0+):

else
local rl = string.len(response)
local header = math.decimaltobyte(129)
if (rl < 126) then
header = header..math.decimaltobyte(rl)
elseif (rl >=126) then
header = header..math.decimaltobyte(126)..math.decimaltobyt e((rl-math.mod(rl,256))/256)..math.decimaltobyte(math.mod(rl,256))
end
response = header..response
end