File upload/download

rfernandez

Hi,

I'm working with r3 3.11.2.6, and at this moment locally in my machine.

I have the following use case:

I need to allow the application to upload/download files (text, binary, and other specialized formats).
The files could be very big, up to tens of Gb or more.
I'm aware of the File field type, that will provide and interface to browse my local machine and upload the file to the path specified in the config.json.

Would this be the right way to solve this scenario with r3?

Additionally, I've noticed that the file names are encoded when saved in the server, and saved in separate folders named as the first 3 characters of the file name.
However in the Form the original file name is shown.
I need to be able to, given the name shown in the Form, provide the encoded name, so in case a user needs to open the file in the server, from outside the application, with another specialized application, he knows which encoded name he must open. So... two questions here:

1. In which table/tables can I find the relationship between these two names? And how would you suggest I provide this association to the user, in the context of r3?
2. Is there a way to override the one file per folder behavior by means of configuration? (I guess there is a purpose behind this way of storing the files, but for my use case is extremely impractical)

And finally, when I edit a record and delete a file associated to the record in the Form, the file is not deleted in the file system. Neither when I delete the whole record. Could this be a bug? or it is the expected behavior?

Thank you!!

gabriel

Uploaded files are not meant to be interacted with on the server. The file storage is managed by r3 for things like delivery (via browser or r3 client), retention and versioning - interacting with the file storage will likely break something.

The "encoding" you mention is just a renaming to internal IDs, with files not being one-to-one representation of files that you upload, but hardlinked file versions. If you upload a 10gig file to r3 and link it in multiple places, only 1 version of that file is being kept. If you "delete" it in one referenced place, the file version is kept, as long as one reference still exists (s. hardlinks).

To your questions directly:

rfernandez 1. In which table/tables can I find the relationship between these two names? And how would you suggest I provide this association to the user, in the context of r3?

If you need to get uploaded files out of r3 from the server, you can use backend instance function calls for file management - in your case primarily file_export()

rfernandez 2. Is there a way to override the one file per folder behavior by means of configuration? (I guess there is a purpose behind this way of storing the files, but for my use case is extremely impractical)

No there is not. When using the file_export() call (s. above) you can define a name and export path. But be aware that you need to define an export path in your config.json file. If you upgraded from r3 3.10 or older these lines might not exist in your file and you must create them - from the announcement:

Due to security concerns, file handling calls are limited to pre-defined paths; these are set in the config.json file via two new parameters: paths->fileImport & paths->fileExport.

rfernandez And finally, when I edit a record and delete a file associated to the record in the Form, the file is not deleted in the file system. Neither when I delete the whole record. Could this be a bug? or it is the expected behavior?

It depends. File retention is complex as there are multiple conditions to meet, before the scheduler cleans up the files. To begin with, the scheduled system job "Cleanup expired file uploads" must run for files to be deleted - by default it runs once per day.

Besides the job needing to run for file cleanup, file retention rules also play a role (s. Admin panel -> Files). And at last if there is still a reference to the file in another record, the file version is being kept.

rfernandez

Hi Gabriel,

I understand, the expected use is to load and download through the application. That's fine.
Noted: do not touch files uploaded with the app from outside the app.

Nice tip the use of the file_export!!

FINDING: If i delete the image I uploaded in a record of a relation using the Form, the corresponding Clean expired uploaded files job does not delete the file in the filesystem...
It only does when the record to which the file belonged to is deleted in the Form.
This could be a problem, because unless I delete the record to which I originally uploaded the file, the file will remain in the filesystem forever. And there is no way to access it anymore, so I cannot see the point in keeping it there.

Could you please take a look at this? In case it is a bug. Otherwise, please confirm this is how it is intended to work.

Thank you!!

gabriel

rfernandez Could you please take a look at this? In case it is a bug. Otherwise, please confirm this is how it is intended to work.

A file that was deleted from a still existing record, must fulfill its retention criteria, before the schedule may delete it. This is to allow for recovery actions to take place.

Did you check your retention settings (s. Admin panel -> Files)? Can you confirm that the deleted file satisfied these conditions and was still not deleted?

rfernandez

Hi Gabriel,

Aha! :-)
I have a setting in Files that says keep deleted files for 90 days.
I guess this is what is preventing the file alone to be deleted. Could it be?

On the other hand, when the record containing the file is delete that conditions is overidden. At least this is the behavior I was able to reproduce.

Thank you!

gabriel

rfernandez I guess this is what is preventing the file alone to be deleted. Could it be?

Yes.

rfernandez On the other hand, when the record containing the file is delete that conditions is overidden.

If the record, that the file is associated with, is deleted, there is nowhere to recover to - so no need to keep any old versions for recovery. That is at least the idea behind the current implementation.