Get a url as a string?

[danward79]

Hi,

Trying to get a url as a variable containing the html as a string.

Unfortunately I am not doing very well. I was trying to copy Marcels way of grabbing data out of a webpage.


local url ="http://www.dilbert.com/";

TEMPHTML = socket.http.get(url);

Marcel uses the code above, but that does not work.

I have tried th following


local url = socket.http.request("http://www.ndbc.noaa.gov/station_page.php?station=62305&unit=M&tz=GMT")
print (url)

This does not work either.

I am hoping to be able to capture some wave data out of the html code.

Any ideas would me most apprieciated

[mhwlng]

look at my girder 4 amg sample

http://www.promixis.com/phpBB2/viewtopic.php?t=11557

it shows how to use http.request


Marcel

[danward79]

Cheers, I knew I saw it somewhere

:lol:

[danward79]

Marcel, looking at your code...

This should work.


local url = socket.http.request("http://www.ndbc.noaa.gov/station_page.php?station=62305&unit=M&tz=GMT")
print (url)

as per


local ARTISTBIOHTML,HTTPERROR = http.request(URL..biourl);

or am I missing something? :oops:

[mhwlng]

not quite....

use something like this


local http = require("socket.http")

local HTMLDATA ,HTTPERROR = http.request("http://www.ndbc.noaa.gov/station_page.php?station=62305&unit=M&tz=GMT");
if ((HTTPERROR == 200) and (HTMLDATA ~= nil)) then
...
...
end

Marcel

[danward79]

Thanks,

It is not me then, it does not work from girder.

However

If I use this...


HTMLDATA ,HTTPERROR = socket.http.get("http://www.ndbc.noaa.gov/station_page.php?station=62305&unit=M&tz=GMT");

print (HTMLDATA, HTMLERROR);

it works from the NR.

I also tried http://www.dilbert.com/, and it did not work.

Could you try this see if it works on your pc, please?


HTMLDATA ,HTTPERROR = http.request("http://www.dilbert.com/");
if ((HTTPERROR == 200) and (HTMLDATA ~= nil)) then
print (HTMLDATA, HTMLERROR);
end

[mhwlng]

works fine for me

you did add this to the top of your lua file, right ???

local http = require("socket.http")


Marcel

[danward79]

yep,

[mhwlng]

you must be missing some lua files in the luascript or luascript\socket directory....
I suggest deleting the girder directory and reinstalling


Marcel

[mhwlng]

we are talking about the latest girder 4, right ??? (since this is in the girder 4 forum)

Marcel

[danward79]

Great minds think alike, I am just trying that now.

[danward79]

we are talking about the latest girder 4, right ??? (since this is in the girder 4 forum)

Marcel

Yep

[mhwlng]

I just add a scripting action


local http = require("socket.http")
HTMLDATA ,HTTPERROR = http.request("http://www.dilbert.com/");
if ((HTTPERROR == 200) and (HTMLDATA ~= nil)) then
print (HTMLDATA, HTTPERROR);
end

note the 'require' bit

and press 'apply' then 'apply and test'

(just pressing F5 doesn't work, I think that this is a bug)

Marcel

[mhwlng]

by the way, you print HTMLERROR instead of HTTPERROR, but the sample still works...

Marcel

[danward79]

Seems to work like this

http://www.ndbc.noaa.gov/station_page.php

but as soon as I add

http://www.ndbc.noaa.gov/station_page.php?station=62305

it does not work

likewise

http://www.dilbert.com/

does not work, but http://www.google.com/ and promixis do!

[danward79]

I just add a scripting action


local http = require("socket.http")
HTMLDATA ,HTTPERROR = http.request("http://www.dilbert.com/");
if ((HTTPERROR == 200) and (HTMLDATA ~= nil)) then
print (HTMLDATA, HTTPERROR);
end

note the 'require' bit

and press 'apply' then 'apply and test'

(just pressing F5 doesn't work, I think that this is a bug)

Marcel

That is how I am doing it. Sorry I missed this post!

[mhwlng]

they all work for me, but I'm on a 8 Mbit ADSL line...

have a look at the httperror

my guess is that data for the non-working url's is returned too slow and the request times out

Marcel

[danward79]

The error being returned is a time out, It seems to be as soon as I add a station id to the url


http://www.ndbc.noaa.gov/station_page.php?station=

is ok

until I put 62305 on it

[mhwlng]

in http.lua I see :

TIMEOUT = 60

you don't happen to accidentally overwrite this with a different value somewhere in your gml file ?

Marcel

[danward79]

No just double checked it is still 60 and the port is 80, which explains why it takes an age to return the error,

Funny how it works fin in NR and not in G4.

[mhwlng]

in NR the http stuff is linked into the NR application, in Girder it's in external lua files. so we can't compare versions etc...

I don't have any other ideas at the moment....

Marcel

[danward79]

Ah right,

Thanks... You better go put your "Thinking Cap" on! :D

[mhwlng]

just an idea

try temporarily removing all stuff from the luascript\startup folder and restarting girder

the weather stuff also uses http.request

there might be a threading issue or something ?

Marcel

[danward79]

No Made no difference what so ever.

[mhwlng]

you can try this :


local TEMPHTML = ""
local url ="http://www.dilbert.com/"
local objXMLHTTP = luacom.CreateObject("Microsoft.XMLHTTP")
objXMLHTTP:Open ("GET",url , 0)
objXMLHTTP:Send()
TEMPHTML = objXMLHTTP.responseBody
objXMLHTTP = nil
collectgarbage()

print (TEMPHTML)


Marcel

[danward79]

Hi Marcel,

Thanks for that it works, but I would rather use the lua version.

In the meantime thou I will be using it.

:)

[danward79]

Marcel

Thanks for the help here is th result

http://www.promixis.com/phpBB2/viewtopic.php?t=11815

:D