Binary File uploads

Hi All,
I’ve got a very simple but apparently complex problem!
I’m hoping someone has a solution for me.
I’ve got an Android app that records video and audio to the Internal Storage. (root folder and DCIM folders)
I need to be able to send those files back to php script on a server at a time when the user has internet connectivity.
In practice that might mean 4,5 or 6 files to be sent back at some time up to a few days after the video and audio was recorded.
The issue is like this:-
If I use a ‘File Chooser’ control to choose the file then upload, all seems to go fine. (this method is not practical for a large workforce where the send needs to be ‘automatic’)
Therefore I need the system to record the filenames as the files are being created and wait for the right time to send back.
No matter what approach I’ve tried (Cordova FileTransfer plugin, XHR etc) I have had no success.
Does anyone have a sample project or code that works ?
I’ve tried meddling with Config files, Contnet Security policies, different methodologies etc, but in general the issue seems to centre on the system not being able to ‘find’ the file on the app to send…even though I have some code that verifies that the file exists.

Unlike photos, which I store in the SQLite DB, it’s not feasible to store Base64 data relating to video due to size constraints.

If anyone has a way to do this or is interested in developing the code to do this, I’d really appreciate the help!
Thanks,
Neil

Neil, I’m not sure I understand what you mean by no success. No success in your app finding the file?, or waiting for the appropriate time? or some or all files don’t upload? or ?

Hi Gary,
In effect I’ve been getting Error Code 1 which translates to ‘local file not found or accessible’.
This is despite using another piece of code to verify that the file does exist…
Thanks,
Neil

Hi,
As an update, I believe I can now transfer file data from Android device to server, but the files on the server seem to be corrupt or incomplete. They seem to be within a few kilobytes of the ‘full’ file but aren’t quite right and are unreadable as video files(MP4). E.g. 9.5MB video files are out by 4KB.
Any ideas ?
I’m sending directly as files here using xhr post.
Thanks,
Neil

Ok…ultimately…I’ve come to the conclusion that what I need to work is not actually possible…
To send data via the latest techniques (rather than deprecated systems like FileTransfer) I must use XHR. To use XHR , i must use FileReader. But in using FileReader… I’m stuck because of the below (from the FileReader Wiki in appstudio) :-

"The FileReader object allows apps to read files into memory. Depending on the platform, files can be flat text, photos, movies or other types of files. Certain platforms restrict which files can be opened. Photo Libraries, DropBox, iCloud Drive and Google drive files can usually be opened.

The file to be opened has to be selected by the user from a TextBox control, with inputType set to File. It is not possible to hard code a file name: this is for security purposes. Otherwise, web apps would be able to open files on user’s systems that should not be allowed."

Due to this: “has to be selected by the user from a TextBox control” I simply can’t see a way to record multiple videos and send them back automatically later on…

Only remaining option would be to, perhaps, process the video on record to save into multiple small B64 records maybe 500KB at a time and then send them back in pieces for the server to reattach later on ?

Very frustrating…
Thanks,
Neil

I have a similar problem on my roadmap and what I’m thinking is webrtc to a cURL endpoint. Would that approach work for you?

Hi,
I can’t say I know much about webrtc, but the ‘real time’ nature of it suggests it’s not the kind of solution i need.
I’m looking for an asychnronous method to ‘poll back’ video clips as and when the user gets to an area of 4G/5G or wifi.
The app user will be recording multiple clips in areas where there is no decent data connection, and so will need to save the video data either as a file or perhaps Base64 data into multiple SQLite fields/records.
I’m running out of options now, it seems…
Thanks,
Neil

Ok, so I’ve done a lot of that kind of stuff in the past so let’s see if I can get caught up on this thread and help you forward.

Server Questions’
What is your server side running? LAMP? If so, then I have some PHP code that receives the uploaded files and some insight there. Something else and I won’t be of much help.

App Questions
I take it that you’re working on a mobile app that will go in the app stores. If that’s correct then this is ultimately a solvable problem. What file plugins have you tried?

Hi,
That sounds promising! :slight_smile:

I’m finished for the day here, but will post some detail of what I tried tomorrow.
Thanks,
Neil

Hi PPetree,
Thanks for trying to help me out!
Here are my answers to the below:-

I’ve been using the file plugin, and the filetransfer plugin.
I’ve had success with the filechooser control uploading mp4s to the server.
What I can’t do is upload files automatically.
Specifically record filename1.mp4, filename2.mp4, filename3.mp4 etc and then send them up automatically at a later time.
I don’t mind using the old filetransfer plugin or the XHR2 method as long as I can get it to work!
(willing to pay for your time if you believe it’s something you can achieve!)

Thanks,
Neil

OK, so those file plugins are the right way to start. They work.

First thing is to check your server’s php.ini file and make sure you are allowing the file sizes you’re trying to upload. This alone could account for some of your file being missing.

Here’s some javascript I used. I had to cut some stuff out but it should be enough for you to understand how I was doing this.

var win = function (r) {
    // if r.response status = OK
    // then we have an image we can build a URL from 
    // once that's done we can delete the local image
    // console.log("Win Code = " + r.responseCode);
    // console.log("Response = " + r.response);
    // console.log("Sent = " + r.bytesSent);
}

var fail = function (error)
{
  var code;
  switch(error.code)
  {
    case FileTransferError.FILE_NOT_FOUND_ERR:
      code = "FileTransferError.FILE_NOT_FOUND_ERR";
      break;
    case FileTransferError.INVALID_URL_ERR:
      code = "FileTransferError.INVALID_URL_ERR";
      break;
    case FileTransferError.CONNECTION_ERR:
      code = "FileTransferError.CONNECTION_ERR";
      break;
    case FileTransferError.ABORT_ERR:
      code = "FileTransferError.ABORT_ERR";
      break;
    case FileTransferError.NOT_MODIFIED_ERR:
      code = "FileTransferError.NOT_MODIFIED_ERR";
      break;
    default:
      code = "FileTransferError.UNKNOWN_UNDOCUMENTED_ERR";
      break;    
  }
  
  // console.log("An upload error has occurred: Code = " + code +" : " +error.code);
  // console.log("upload error source " + error.source);
  // console.log("upload error target " + error.target);
}

// userid
// video
// type = crime || weather || accident)
// eventid
// entryid
function upload(userID, video, type, eventID, itemID)
{
  var tVideo = extractFileName(video); 
  
  window.resolveLocalFileSystemURL(cordova.file.documentsDirectory, function (dir) {
      dir.getFile(tVideo, {create: false}, function (fileEntry) {
          var fileURL = fileEntry.nativeURL;

          var options = new FileUploadOptions();          
          options.fileKey = "file";
          options.fileName = tVideo;
          // you can code and figure out which to use
          options.mimeType = "video/mp4";                 // .mp4, .m4a, .m4p, .m4b, .m4r, .m4v
          options.mimeType = "video/mpeg";
          
          var params = {};
          params.userid = userID;
          params.type = type;         // crime/weather/accident
          if(eventID)
          {
            params.eventid = eventID;   // event id
          }
          if(item)
          {
            params.itemid = itemID;   // item id
          }
          options.params = params;
          
          var ft = new FileTransfer();
          // serviceURL = https://something.domain.com/
          ft.upload(fileURL, encodeURI(serviceURL +"domainImageUpload.php"), win, fail, options);
      }, onErrorCreateFile);
  }, onErrorLoadFs);
}

function extractFileName(file)
{
  var pieces = file.split('/');
  var tVideo = pieces[pieces.length-1];
 
  tVideo = "videos/" +tVideo.split('?')[0];       // trimmed photo 
  return(tVideo);
}

Here’s the php on the server side. Again, I had to strip a bunch of stuff out but there’s enough comments and debugging stuff to show you what I was doing.

<?php

$upload = $_FILES['file'];      // the file package
$userid = $_POST['userid'];
$eventid = $_POST['eventid'];
$itemid = $_POST['itemid'];
$purpose = $_POST['type'];      // weather / crime/ accident etc

$max_size = 24000000;           // (24mb) define the maximum size we'll accept for a video


// output the header
header('Content-type: application/json');  // use this and iframe wraps <pre></pre> around our json
// header('Content-type: text/html');  // use this since we're returning to an iFrame

// log the $_FILE input structure so we can see whats going on
// error_log("PHP File Info: " .print_r($_FILES, true));

//Check that we have a file
if( (!empty($upload['tmp_name'])) && ($upload['error'] == 0) )
{
  $img_info = getimagesize($upload['tmp_name']);
  // error_log("Image Info: " .print_r($img_info, true));
 
  // now make sure it's a valid/accepted file type
  switch($upload["type"])
  {
    case "image/gif":
    case "image/pjpeg":
    case "image/jpeg":
      if($img_info['bits'] >= 8)
        break;

   case "video/mp4":
   case "video/mpeg":
        break;
      
    default:
      $response = Array('status' => "ERR", 'text' => "Error: " .$_FILE['FILE']["type"] ." is not a supported file type.");   
      $output = json_encode($response);
      echo $output;
      exit;
  }
  
  // now make sure it's an acceptable size  
  if($upload["size"] < $max_size) 
  {
      // Determine the path to which we want to save this file
      // $newname = dirname(__FILE__).'/images/'.$filename;
      
      $file_info = buildNewFileName($filename, $userid, $purpose);

      //Check if the file with the same name is already exists on the server
      // error_log("checking if file exist" .$file_info['save']);
      if (!file_exists($file_info['save']))
      {
        // error_log("File didn't exist!");        
        //Attempt to move the uploaded file to it's new place
        if ((move_uploaded_file($upload['tmp_name'], $file_info['save'])))
        {
          // tell the database where this file is stored
          $rc = updateDatabase(....);
          if($rc)
          {
            $response = Array('status' => "OK", 'text' => $file_info['url']);   
            // error_log("It's done! The file has been saved as: ".$newname);
          }
          else
          {
            $response = Array('status' => "ERR", 'text' => "Error: A problem occurred updating your record!");          
          }
        }
        else
        {
           $response = Array('status' => "ERR", 'text' => "Error: A problem occurred during file upload!");   
           // error_log("Error: A problem occurred during file upload!");
        }
      }
      else
      {
         $response = Array('status' => "ERR", 'text' => "Error: File " .$upload['name'] ." already exists");   
         // error_log("Error: File " .$_FILES["uploaded_file"]["name"] ." already exists");
      }
  }
  else
  {
     $response = Array('status' => "ERR", 'text' => "Error: Only videos under $max_size Kb are accepted for upload");   
     // error_log("Error: Only images under $max_size" ."Kb are accepted for upload");
  }
}
else
{
  $response = Array('status' => "ERR", 'text' => "Error: No file uploaded");   
  // error_log("Error: No file uploaded");
}

$output = json_encode($response);
// error_log("json sent to browser: $output");
echo $output;
?>

HTH!

Oh, as for doing the “online” uploads.

I stored the video names in the sqlLite table as an upload queue.

When a video was recorded and put in the upload queue, I did a quick little update on the server and pushed a notice out to the user that the file had been queued for upload the next time they were on wifi. I would remind them to connect daily via a push notification.

When the app was started I checked the connection type and if it was wifi, I looked through the table for the oldest video first and uploaded it.

My app was used by field engineers and they would video equipment in remote places. Sometimes it would be days before they could get back to a signal. Most were good at remembering to let the app sync, a few were not and it’s the few you have to code for. I had several who would forget they were uploading and throw the phone in their pocket and go to lunch. shrug

Hi PPetree,
Thanks very much for your help! I’m on it now and will see if I can get it up and running today on my systems here! Will keep you posted.

Hi again!
I’m struggling to get this working.
Can I clarify if it’s essential that I store the mp4 in a particular place on the Android device?
I’m trying to upload from the Internal Storage root directory…(or anywhere else to be honest)
Also… is there anything in Content Security Policy and / or the Config file that I must have?
(i do have the file plugin and the filetransfer plugins in place)

I can’t seem to get to the upload phase as yet…
Error : Uncaught typeError: Wrong type for parameter “uri” of resolveLocalFileSystemURI: Expected String, but got Null… line 425 column 9 (of phonegap.js)

Thanks,
Neil

Trust me when I tell you I get how frustrating this is… I spent weeks trying to figure it out.

The upload isn’t working because URI isn’t filled in (null). Using the camera plugin, you can allow users to pick a video from their gallery. There’s a setting that specifies the file type be returned as a URI. See this example (destinationType):

    var options = {};
        options.sourceType = Camera.PictureSourceType.PHOTOLIBRARY;
        options.quality = Number(getPhotoQuality());
        options.destinationType = Camera.DestinationType.FILE_URI;
        options.correctOrientation = true; 
      
    navigator.camera.getPicture(function(imageData) {  
          that.cameraOnSuccess(imageData);
        }, function(message) {
          that.cameraOnFail(message);
        }, options);

The above example is for photos but that setting is key. If memory serves me correctly, once you have that URI you can upload it.

Ok…so you can’t send a file unless you’ve created it in the app and stored the FileUri?

This is the crux of my problem, I think. How do I take that FileURI, store it in SQLite and then refer to it and send back?

( I think I’m ok with the sending bit,it seems; it’s the part before that! )

Ok, so I’m now recording video and afterwards saving the full path to the textbox as shown.
Is that the FileURL that’s needed?

Here’s the latest project, if you could take a look if possible and see what’s wrong when you get a chance.
videouploadsolution.zip (11.8 KB)

Yes, that’s the file URI. Is the upload rejecting that?

It would be Monday before I can look at the code. I’m on a REALLY serious deadline at the moment.

Monday will be fine! It’s been a few weeks of frustration for me, so a few more days won’t be a problem!
Thanks again