PDA

View Full Version : Is it possible to make webserver play nice with special characters in variables?



Mastiff
April 30th, 2015, 03:46 AM
I have a problem with the webserver: When there are variables that contains special characters, like (Norwegian), at least the (which is the one I'm using so far) is shown like a �. So a variable that should have been Kjkken (kitchen) is instead shown as Kj�kken. Is it possible to get this to work? I have the web page in UTF-8, and stuff that is written in regular text works. But variables doesn't want to play nice. I tested with the demo that comes with Girder, and it seems like it's a problem within the webserver or somewhere between that and the generated webpage. Is it possible to get this working as it should?

Ron
April 30th, 2015, 07:26 AM
You are most likely not using the correct encoding. If it's displaying as a �. Check the headers that are returned and fix those or add a meta tag:

http://www.w3schools.com/tags/att_meta_charset.asp

Probably UTF-8 is incorrect, I assume your text editor is not creating proper UTF-8 files.

http://scratchpad.wikia.com/wiki/Character_Encoding_Recommendation_for_Languages

Shows that ISO-8859-1 might work. Place this at the top of your HTML file inside the <HEAD> tags.



<meta charset="ISO-8859-1">

Mastiff
April 30th, 2015, 07:58 AM
Thanks, Ron! But I'm afraid that didn't work. When I use that (or the -15 which is on the same page), it doesn't change the at all. Plain text works as it should, it's just those variables that won't work. I have tried with both notepad, Dreamweaver and Notepad++, all set to UTF-8, and I get the same result from them all.

The start of my file look like this, with the charset changed:


<html><head>
<meta http-equiv="PRAGMA" content="NO-CACHE">
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>Temperaturstyring</title>

That works without problems, except for the carbled variable for . If I do this:


<html><head>
<meta charset="ISO-8859-1">
<meta http-equiv="PRAGMA" content="NO-CACHE">
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>Temperaturstyring</title>

I get exactly the same result. Even deleting the one after content doesn't change anything.

Ron
April 30th, 2015, 08:03 AM
Show us the file!

Mastiff
April 30th, 2015, 08:06 AM
With pleasure, sir! :) I've removed your meta again for now since it didn't seem to change anything. It's attached as a text file, since it's too long for the message.

Ron
April 30th, 2015, 08:11 AM
which line creates the offending characters?

Mastiff
April 30th, 2015, 08:14 AM
//as above, but with letters (does not update unless page is reloaded)
var myname_REF = '<center><Div><Span id=\"mynameID'+RowID+'\"></span></center>';
$('<td style=\"text-align: center; vertical-align: middle; width: 110px; height: 36px;\">'+myname_REF+'</td>').appendTo(tr);

It takes the room names from this table:


Temperaturstyring.Nettside.Rom ={
[1] = "WC",
[2] = "Garderoberom",
[3] = "Midtsoverom",
[4] = "Hjrnesoverom",
[5] = "Hovedsoverom",
[6] = "Bad",
[7] = "Stue",
[8] = "Kjkken",
[9] = "Kino",
[10] = "Biljardrom",
[11] = "Kjellerkontor",
[12] = "Treningsrom",
[13] = "Gjesterom",
}

Edit: The table goes into the variable on the webpage via this line:


$(document).ready(function() { setTimeout('checkGIRDER(\'Temperaturstyring.Nettsi de.Rom\',\'myname\',\'csv\',\'false\')',200); });

Ron
April 30th, 2015, 08:22 AM
Ah ok. It might be the encoding of the ajax-request that is wrong.

ajaxreqjson.lhtml

show us that file.

Mastiff
April 30th, 2015, 08:24 AM
Certainly, boss!


<%


function getfield (f)
local v = _G -- start with the table of globals
for w in string.gfind(f, "[%w_]+") do
if type(v) == 'table' then --v ~= nil
v = v[w]
else
return false
end
end
return v
end

local s,vartype
local t = webserver:GetCGI()

if t.Data then
local json = require('json.json')
local data = string.Split(t.Data,",")
webserver:print('{')
for i in data do
if (i~=1) then
webserver:print(',')
end
webserver:print('"' .. data[i] .. '":')
s=getfield(data[i])
if s then
vartype = type(s)
if ( vartype =='function' ) or ( vartype =='thread' ) or ( vartype == 'userdata' ) then
webserver:print('"[' .. vartype .. ']"')
else
if (s==nil) then
webserver:print('null')
else
webserver:print(json.encode(s))
end
end
else
webserver:print('"[Variable Not found]"')
end
end
webserver:print('}')
end



%>

Ron
April 30th, 2015, 08:32 AM
Try adding



webserver:setHeaderEx("Content-Type", "application/json; charset=ISO-8859-1", true);


after the initial <%

Next I'd use FireFox developer console to see exactly what is being transfered (F12 -> Network )

Mastiff
April 30th, 2015, 08:46 AM
That line killed everything. The developer console says:


[string "WebServer"]:2: attempt to call method `setHeaderEx' (a nil value)

Is it possible this is for Girder 6?

But when I look at the developer console and check response for the correct call, I see that the "Temperaturstyring.Nettside.Rom:Array" has "Hjrnesoverom" as number three and "Kjkken" as number 7, so it's correct that far.

Ron
April 30th, 2015, 08:52 AM
No that was Girder 5.1 code. Try:



webserver:setHeader("Content-Type: application/json; charset=ISO-8859-1", true);

Mastiff
April 30th, 2015, 08:56 AM
Console:


attempt to call method `setHeader' (a nil value)

But there are still no error messages in Girder.

Ron
April 30th, 2015, 09:04 AM
show us the ajax file after the modification.

Mastiff
April 30th, 2015, 09:07 AM
<%
webserver:SetHeaderEx("Content-Type: application/json; charset=ISO-8859-1", true);

function getfield (f)
local v = _G -- start with the table of globals
for w in string.gfind(f, "[%w_]+") do
if type(v) == 'table' then --v ~= nil
v = v[w]
else
return false
end
end
return v
end

local s,vartype
local t = webserver:GetCGI()

if t.Data then
local json = require('json.json')
local data = string.Split(t.Data,",")
webserver:print('{')
for i in data do
if (i~=1) then
webserver:print(',')
end
webserver:print('"' .. data[i] .. '":')
s=getfield(data[i])
if s then
vartype = type(s)
if ( vartype =='function' ) or ( vartype =='thread' ) or ( vartype == 'userdata' ) then
webserver:print('"[' .. vartype .. ']"')
else
if (s==nil) then
webserver:print('null')
else
webserver:print(json.encode(s))
end
end
else
webserver:print('"[Variable Not found]"')
end
end
webserver:print('}')
end



%>

I think maybe it should be SetHeaderEx. That does a job, but can't handle the boolean true/false that's used in the script:


[string "WebServer"]:2: bad argument #2 to `SetHeaderEx' (string expected, got boolean)
stack traceback:

The SetHeader doesn't give anything. But I see an error here, it can't find the CSS file:


<HTML><HEAD><TITLE>404 Not Found</TITLE></HEAD><BODY><H1>Not Found</H1><P>The Requested URL /styles.css was not found on this server.</P><HR><i>Promixis Girder Webserver 3.0.4 (5.1.1) Thu, 30 Apr 2015 13:06:24 GMT</i></BODY></HTML>

Can that be related?

Ron
April 30th, 2015, 09:11 AM
Try

<%
webserver:SetHeader("Content-Type: application/json; charset=ISO-8859-1", true);

Mastiff
April 30th, 2015, 09:15 AM
I think that's the same as one of the first we tried. It kills all the variables and gives this response instead of the list of rooms:


%
webserver:SetHeader("Content-Type: application/json; charset=ISO-8859-1", true);

function getfield (f)
local v = _G -- start with the table of globals
for w in string.gfind(f, "[%w_]+") do
if type(v) == 'table' then --v ~= nil
v = v[w]
else
return false
end
end
return v
end

local s,vartype
local t = webserver:GetCGI()

if t.Data then
local json = require('json.json')
local data = string.Split(t.Data,",")
webserver:print('{')
for i in data do
if (i~=1) then
webserver:print(',')
end
webserver:print('"' .. data[i] .. '":')
s=getfield(data[i])
if s then
vartype = type(s)
if ( vartype =='function' ) or ( vartype =='thread' ) or ( vartype == 'userdata' ) then
webserver:print('"[' .. vartype .. ']"')
else
if (s==nil) then
webserver:print('null')
else
webserver:print(json.encode(s))
end
end
else
webserver:print('"[Variable Not found]"')
end
end
webserver:print('}')
end



%>

Ron
April 30th, 2015, 09:20 AM
no we had a small "s" first. That was wrong.

Mastiff
April 30th, 2015, 09:20 AM
Actually that seems to be what's returned for all fields.

Mastiff
April 30th, 2015, 09:23 AM
Yeah, right. Sorry. Still it seems to only send the contents of the rqjson file.

Ron
April 30th, 2015, 09:30 AM
post what it sends here.

Mastiff
April 30th, 2015, 09:32 AM
If you mean in the "Resonse" part of the console, I did on the previous page. It sends the contents of the file ajaxreqjson.lhtml over and over again.

Ron
April 30th, 2015, 09:36 AM
did you delete the first carret? ( for girder to parse that code it must start with <% )

Mastiff
April 30th, 2015, 09:43 AM
Yes, sorry. I was in the process of editing the previous message whith what it actually sends after I had put back the < but it was too long to fit in the post. I have attached it here. It sends on jquery.min.js and jquery-1.5.2.min.js, response is in the zip files. Even as text files it was too large for attached text files! Also it doesn't loop like it's supposed to do.

Ron
April 30th, 2015, 09:47 AM
Sorry you have completely lost me. Surely it's not sending jquery.

Mastiff
April 30th, 2015, 09:53 AM
My bad, I see what you mean, yeah. That stuff's sent every time, in the beginning. With the line webserver:SetHeader("Content-Type: application/json; charset=ISO-8859-1", true); added there just isn't anything after those two jquery lines. Without that line it goes on sending the ajax stuff.

Mastiff
April 30th, 2015, 10:09 AM
It seems like the problem is deeper in the webserver code. You can try yourself, if you want to. Set a variable with the letters "" as the value (just copy them from this post), and go to the standard pages of the Girder Webserver. Then you can go to the scripting part and type in the value name in the field. On my system I then get �� as the result (one of them simply dissappears). If you do a printout of the variable within Girder you will probably see something else since you have English Windows, but my Lua Console shows .

Ron
April 30th, 2015, 10:12 AM
I'm not convinced it's in the webserver code. Character encoding isn't easy. You need to make sure that the encoding that you have in the file matches the encoding that you are displaying EXACTLY. Clearly you have a mismatch.

Mastiff
April 30th, 2015, 10:33 AM
Sorry, I apologize, you're right (not a shocker, I guess...). Something is messing up somewhere. I went very basic. I took the script part from the script demo, saved it in Notepad as an UTF-8 file (it was ANSI) and saved it in the standard directory of the webserver demo. That gave me the garbled . Then I added your line to the ajaxreq.lhtml file, and it works! like a champ! So that line works in the ajaxreq.lhtml in the demo, but not in the ajaxreqjson.lhtml in my actual setup. This is the original ajaxreq.lhtml where that line works:


<%
webserver:SetHeader("Content-Type: application/json; charset=ISO-8859-1", true);

function getfield (f)
local v = _G -- start with the table of globals
for w in string.gfind(f, "[%w_]+") do
v = v[w]
end
return v
end

local s,vartype
local t = webserver:GetCGI()

if t.Data then
s=getfield(t.Data)
if s then
vartype = type(s)
if ( vartype == 'table' ) or ( vartype =='function' ) or ( vartype =='thread' ) or ( vartype == 'userdata' ) then
webserver:print('['..vartype..']');
else
webserver:print(s)
end
else
webserver:print('[Variable Not found]')
end
end



%>

This is the ajaxreqjson.lhtml where the same line doesn't work because it stops processing:


<%
webserver:SetHeader("Content-Type: application/json; charset=ISO-8859-1", true);

function getfield (f)
local v = _G -- start with the table of globals
for w in string.gfind(f, "[%w_]+") do
if type(v) == 'table' then --v ~= nil
v = v[w]
else
return false
end
end
return v
end

local s,vartype
local t = webserver:GetCGI()

if t.Data then
local json = require('json.json')
local data = string.Split(t.Data,",")
webserver:print('{')
for i in data do
if (i~=1) then
webserver:print(',')
end
webserver:print('"' .. data[i] .. '":')
s=getfield(data[i])
if s then
vartype = type(s)
if ( vartype =='function' ) or ( vartype =='thread' ) or ( vartype == 'userdata' ) then
webserver:print('"[' .. vartype .. ']"')
else
if (s==nil) then
webserver:print('null')
else
webserver:print(json.encode(s))
end
end
else
webserver:print('"[Variable Not found]"')
end
end
webserver:print('}')
end



%>

Are you able to see what's going on there? It seems to my untrained eye that the main difference is the table stuff.

able

Mastiff
May 10th, 2015, 06:20 AM
Sorry if I'm pestering you, Ron. But do you see what in the second script that makes it stop processing the table when using the line that sends the correct coding? I aske Jon, and he thought it could be the json. So does that mean that it's impossible to get this right here? :confused:

Ron
May 10th, 2015, 08:10 AM
Do you get a Lua/Logger error? Can you show me the output that you do get?

Mastiff
May 10th, 2015, 08:22 AM
Hi, Ron! No, I don't. There's no unusual output in Girder, the log display shows the same stuff as when I'm not using this line (it's constantly refreshing to reflect changes from clicking or entering temperatures). This is with the line:


Time Date Source Details Payloads
14:12:47:954 5/10/2015 WebServer /ajaxreqjson.lhtml Data=myVal,Temperaturstyring.Nettside.Termostatmod us.0 _=1431259970364
14:12:47:959 5/10/2015 WebServer /ajaxreqjson.lhtml Data=myVal,Temperaturstyring.Nettside.Termostatmod us.1 _=1431259970367
14:12:47:964 5/10/2015 WebServer /ajaxreqjson.lhtml Data=myVal,Temperaturstyring.Nettside.Termostatmod us.2 _=1431259970369



This is without it:

Time Date Source Details Payloads
14:14:48:658 5/10/2015 WebServer /ajaxreqjson.lhtml Data=myVal,Temperaturstyring.Nettside.Termostatmod us.0 _=1431260090787
14:14:48:664 5/10/2015 WebServer /ajaxreqjson.lhtml Data=myVal,Temperaturstyring.Nettside.Termostatmod us.1 _=1431260090789


The difference is in what I see. If I click on one of the (invisible) buttons the reaction is what I get as well:


Time Date Source Details Payloads
14:15:54:090 5/10/2015 WebServer /ajax_sendevent_payload.lhtml device=18 event=SettTermostatmodus payload1=1 payload2 payload3=1 payload4=FromWeb
14:15:54:094 5/10/2015 Girder SettTermostatmodus 1 1 FromWeb


This tells girder to set room number 1 (payload1) to day temp (payload3), and that this comes from the webpage, and not a timer (payload4). Here are screenshots of the difference.

Ron
May 10th, 2015, 08:22 AM
Try this one: ( I didn't try on Girder 5, we're about to head out, it's mothersday after all. )



<%
webserver:SetHeader("Content-Type: application/json; charset=ISO-8859-1", true);

function getfield (f)
local v = _G -- start with the table of globals
for w in string.gfind(f, "[%w_]+") do
if type(v) == 'table' then --v ~= nil
v = v[w]
else
return false
end
end
return v
end

local s,vartype
local t = webserver:GetCGI()

if t.Data then
local json = require('json.json')
local data = string.Split(t.Data,",")
webserver:print('{')
for i in ipairs(data) do
if (i~=1) then
webserver:print(',')
end
webserver:print('"' .. data[i] .. '":')
s=getfield(data[i])
if s then
vartype = type(s)
if ( vartype =='function' ) or ( vartype =='thread' ) or ( vartype == 'userdata' ) then
webserver:print('"[' .. vartype .. ']"')
else
if (s==nil) then
webserver:print('null')
else
webserver:print(json.encode(s))
end
end
else
webserver:print('"[Variable Not found]"')
end
end
webserver:print('}')
end



%>

Mastiff
May 10th, 2015, 08:25 AM
Nope, sorry. Same thing. :( As for mother's day, I didn't know the US had another day then us. ;) We have it the second Sunday in February. And it's not a big hurry, but I would appreciate it if you could have a look at it when you get the time! It's sort of the only remaining thing on that temperature control system, at least the only thing I know about!

Mastiff
December 9th, 2015, 12:03 PM
I'm dragging zombie up again here, but I have from time to time thought about asking again, and since it's only five months unil til system should be totally glitch free because I'm putting the house up for sale, I thought I'd ask if you have any ideas. Everything is still as before. Execution of the main script just stops by adding the first line of your previous script to the setup. And I have those pesky � instead of here. Do you have any idea what this can be? Please?

Mastiff
December 16th, 2015, 04:27 AM
Finally found it with help from a local web guru. :o This line in the ajaxreqson script fixed it:


webserver:SetHeaderEx("Content-Type", "text/plain; charset=iso-8859-1", true)