How to retrieve a "text" file from a website and display it in a TextArea

falcon · December 24, 2021, 5:59pm

Phil,
Thanks so much. I was able to get the code to work in the browser. But I had rem out these lines to get it to run:

$headers = new Headers;
$headers->sendHeaders("json");

I am assuming I should be sending "new Headers" somehow from my app. How can I do this?

PS, FYI, I was also able to get it to work from the command line with both the PHP file and the Tzones.dat file in the root directory.
Paul

PPetree · December 24, 2021, 6:37pm

You’ll not want to put all those files in your root directory on the server. That will ultimately lead to a complete mess.

Those two header lines is what the server sends back to the app. I can tell you still don’t under stand what’s going on with CORS. Let me try to explain it again.

Your app list the resources it can access. It does this in two places:

config.xml
the content security policy header

<meta http-equiv="Content-Security-Policy" content="default-src 'self' 'unsafe-inline' 'unsafe-eval' filesystem: gap: ws://localhost:* https://*.mydomain.com/ https://maps.googleapis.com/; style-src 'self' 'unsafe-inline' https://*.googleapis.com/; font-src 'self' data: https://fonts.gstatic.com/; script-src 'self' 'unsafe-eval' 'unsafe-inline' https://maps.googleapis.com/; img-src 'self' data: https://*.gstatic.com/ https://*.googleapis.com/; connect-src 'self' https://*.mydomain.com/ ws://localhost:*; media-src 'self';">

When you try to load a resource external to the device, the cordova framework (mandated by iOS and Android) will check the Content-Security-Policy header to make sure that you’re allow to load that resource. The same thing applies when you try to make an ajax call. In your case you would change mydomain.com to ‘quickplanning.com’.

Once the cordova framework has determined that you are allowed to request something from quickplanning.com, anything returned must specify that it is from quickplanning.com or the cordova framework will throw a content-security-policy violation error.

Those headers you commented out should either return what’s needed or, via error_log allow you to see what’s needed.

Here’s some light reading for you on CSP/CORS.

falcon · December 29, 2021, 11:11pm

Thanks Phil. I hope you had a nice Christmas.
Yes, I do create folders to keep my root directory uncluttered. I made your changes to the files you sent and put them in the "folder".
When I run this command ("https://QuickPlanning.com/folder/serGetFile.php) from the browser it returns me to my website main page and I get the following statements in the "error_log" in the "folder" directory. The first 3 statements are from the http file and the last one is from the serGetFile.

[29-Dec-2021 22:24:50 UTC] PHP Notice:  Undefined index: HTTP_ORIGIN in /home/pwleblanc/public_html/folder/httpHeaders.class.php on line 55
[29-Dec-2021 22:24:50 UTC] getting HTTP_ORIGIN: 
[29-Dec-2021 22:24:50 UTC] this->http_origin set to: https://quickplanning.com
[29-Dec-2021 22:24:50 UTC] {"status":"OK","text":"Aviation Specialty Programs\r\n15764 Blank Pike\r\nWapakoneta\r\nOhio\r\n45895\r\n02\/15\/2022\r\n15.00\r\nNA\r\nNA\r\n39 \r\nNA\r\n3.14 \r\n11.06 \r\nCA\r\n12.31 \r\n1.01 \r\nMX\r\n4.04 \r\n10.30"}

The 4th line is what the php file is sending back to the server (and this is the correct data).

When I run it from AppStudio with the following command "req=ReadFile("https://QuickPlanning.com/folder/serGetFile.php")"
I get the following in the "error_log"

[29-Dec-2021 22:59:46 UTC] getting HTTP_ORIGIN: http://127.0.0.1:49243
[29-Dec-2021 22:59:46 UTC] this->http_origin set to: *
[29-Dec-2021 22:59:46 UTC] {"status":"OK","text":"Aviation Specialty Programs\r\n15764 Blank Pike\r\nWapakoneta\r\nOhio\r\n45895\r\n02\/15\/2022\r\n15.00\r\nNA\r\nNA\r\n39 \r\nNA\r\n3.14 \r\n11.06 "}

and the following in the app: (req.err = undefined : req.status = 0)

Do you have any suggestions. I did put your meta file, with changes, into my Project Properties, ContentSecurityPolicy.

Thanks for the extra reading!!!
Paul

PPetree · January 5, 2022, 4:02pm

Hi Paul. Sorry for the slow response… holidays were hell.

In the first example it’s not finding the HTTP_ORIGIN so my guess is the AJAX request isn’t sending it.

In the second example, what’s getting returned by the server is not allowed by the calling page. * means accept everything but you’d have to have * in your content-security-policy definition (apple will not approve apps that do this).

Looks like you need to read that site and make sure your CSP is set correctly and maybe post that if you can’t get it to work.

falcon · January 7, 2022, 12:12am

Hi John,
I’m still trying this to work for me. I have checked my error log and my php file is sending the data, but my app is not reading it. I’d like to start from the beginning with a working model if you can send me some information? I think my problem is somewhere in my csp statement.

You stated you had it working with: “req=ReadFile(filename)”.

Can you send me a copy of the exact contentSecurityPolicy statement you have entered into the Project Properties… contentSecurityPolicy … in appstudio.?
If you are using a php file to send the data, can you send that too? But from your post it looks like you are not using a php.
And any other files I may need to get this working, or statements I need in other areas?

When I get it working I’ll post a working sample for all.
Thank you,
Paul

johnC · January 7, 2022, 1:23am

No contentsecuritypolic
No php
No other files, see below previously posted.

I use the following. Note the randomValue which prevents cached data from being returned. Also I am not testing for req.status = 0

randomValue = SysInfo(10)
C = “version.txt?” & randomValue
req=ReadFile(“[ **https://mydomain.com/deployedbolder/** ](https://mydomain.com/subbolder/)” & C)
If req.status = 200 Then
C = req.responseText

Important: The file to be downloaded must be at or below the deployedfolder. My server is Bluehost. Only tested as a WebApp, not native.

John

ghenne · November 17, 2022, 8:45pm

@falcon asked me to review this topic to see if there is anything we could add.

The replies for the most part are excellent and cover the topic well. We saw it at the time. The topic isn’t really an AppStudio issue - it’s a “How do I program this…” kind of issue, beyond the scope of our support.

A few observations:

The ReadFile function is a wrapper to simplify making an Ajax call. It in turn calls Ajax().
Ajax calls the jQuery Ajax function, so if you really want to go in depth on it, check there.
Once upon a time, getting a txt file from a website was easy, since there weren’t restrictions on it. That allowed all kinds of bad behaviour, such as scraping other company’s websites for data.
CORS restrictions stop that behaviour, so you really have to respect them if you want to read a file.
There’s a lot of new stuff to learn here, with no shortcuts.
There are a lot of resources on the web explaining more.

falcon · December 25, 2022, 9:34pm

Hi guys,
It has been a year since I have beenable to work on this. Still cannot retrieve a text file from remote werver. This last week I have been reading about CORS and things make more sense now. But still unable to read the file.
I have written a simple one page app to test it. I use a simple call:

 btnReadFileRemote.onclick = function() {
  filename = "https://quickplanning.com/folder/serGetFile.php";  
  Textarea1.value = "  ";
  req = ReadFile(filename);
if(req.status==200) {
    Textarea1.value = req.responseText;
  } else {
    Textarea1.value = "status: " + req.status;
  }
};

The file serGetFile.php is supposed to send back the correct response as well as the data. Phil here is the error log on my server:

[25-Dec-2022 20:46:28 UTC] getting HTTP_ORIGIN(on line 56): http://127.0.0.1:58711
[25-Dec-2022 20:46:28 UTC] this->http_origin set to: *
[25-Dec-2022 20:46:28 UTC] HTTP_ORIGIN: *
[25-Dec-2022 20:46:28 UTC] {"status":"OK","text":"Aviation Specialty Programs\r\n15764 Blank Pike\r\nWapakoneta\r\nOhio\r\n45895\r\n02\/15\/2023\r\n15.00 "}

When I run the program I get a status = 0 . and and 2 errors:

jquery3.js:2 Access to XMLHttpRequest at 'https://quickplanning.com/' 
(redirected from 'https://quickplanning.com/folder/serGetFile.php') from origin
 'http://127.0.0.1:58711' has been blocked by CORS policy:
 No 'Access-Control-Allow-Origin' header is present on the requested 
resource.
send @ jquery3.js:2
ajax @ jquery3.js:2
Ajax @ appstudioFunctions.js:431
ReadFile @ appstudioFunctions.js:405
btnReadFileRemote.onclick @ code.js:4


jquery3.js:2  GET https://quickplanning.com/ net::ERR_FAILED 200
send @ jquery3.js:2
ajax @ jquery3.js:2
Ajax @ appstudioFunctions.js:431
ReadFile @ appstudioFunctions.js:405
btnReadFileRemote.onclick @ code.js:4

I know we also have to have the correct statements in the “onfig.xml” file. I have:

  <allow-navigation href="*" />
  <access origin="*" />
  <allow-intent href="*" />

*** So beginning here. Is this correct? Do I need to change anything here?

At this point I am at a loss. Phil, everything you sent me makes sense… I’m just at a loss as to where to go next.

All help is welcome.

Paul

falcon · December 25, 2022, 9:36pm

Hi Phil, I finally got back on this and ;posted a long reply. What you were saying now makes sense, but I still need help. Can you look at this?
Thanks,
Paul

falcon · December 25, 2022, 9:42pm

John, I am finally getting back on this. In the code you sent me can you explain what I would type in the ReadFile request? My data is in the following folder:

https://quickplanning.com/folder/tzones.dat

Here is the code you sent:

randomValue = SysInfo(10)
C = “version.txt?” & randomValue
req=ReadFile(“[ **https://mydomain.com/deployedbolder/** ](https://mydomain.com/subbolder/)” & C)
If req.status = 200 Then
C = req.responseText

Thanks,
Paul

johnC · December 26, 2022, 3:18pm

This is how I might code if my server was hosting quickplanning.com:

randomValue = SysInfo(10)
C = “tzones.dat?” & randomValue
req=ReadFile(“https://quickplanning.com/” & C)
If req.status = 200 Then
C = req.responseText
End If

John

falcon · December 26, 2022, 5:24pm

John,
Thanks for the quick reply. I have tried that and it did not work, but will try again!!
Can you explain why this “readfile” is different from what you sent before?

req=ReadFile(“[ **https://mydomain.com/deployedbolder/** ](https://mydomain.com/subbolder/)” & C)

Why were you using a deployed folder and and a subfolder? Can you explain what is going on in your code?
Paul

johnC · December 26, 2022, 8:31pm

Ignore what was in the brackets […], just remove that stuff, it was for informational purposes.

You can put the file in either your main deployed folder or any folder below (sub folders), your choice. But if you deploy your app to quickplanning.com and you include tzones.dat in the deploy, then just use the deploy folder in the ReadFile as in my example.

John

falcon · January 1, 2023, 5:26pm

Hi Guys,
I wanted to update all with what I have found out to be able to retrieve data from my website.
Basically I did 3 things.

I took Phil’s advice and wrote a .php file on my server and accessed it with a call from my program: req=ReadFile(“https://quickplanning.com/somefolder/some.php”
Then to be able to retrieve the data I placed a “.htaccess” file in “somefolder” (same folder as my php file) which had the statement: #Header Set Access-Control-Allow-Origin * . I’m told if you put this statement at top of php file… you don’t need it here! But I did it anyway.
Then I also put this statement at the top of my ".php " file: header('Access-Control-Allow-Origin: *');

I know it is not the best way, but the only way I could get data from my server and the php can also call data from other servers!!!

Anyway, thanks for all the help,
Paul