and some more...
There are lot of inconsistencies in our ->search calls. We could
simplify some of them, but not in this patch. Here we want to prevent
regressions as much as possible and so don't add unecessary changes.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
It has been missed on bug 17600.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Adjusted commit title.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
.pm must not have -x
.t must have -x
.pl must have -x
Test plan:
Apply only the first patch, run the tests and confirm that the failures
make sense
Apply this patch and confirm that the test now returns green
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Tests were failing with:
# Failed test 'Test delete via UploadedFile as well as UploadedFiles'
# at t/db_dependent/Upload.t line 193.
DBIx::Class::Row::delete(): Not in database at /kohadevbox/koha/Koha/Object.pm line 219
I am not sure this patch is perfect, a set of uploaded files should be
deleted in a transaction, which would be rollback if something is wrong.
But it will be tricky to restore the files after they have been deleted.
It seems that we should deal with that with a more complicated process
and should be part of a separate bug.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
The error codes 1 to 7 are used in Uploader.pm or tools/upload.pl.
It would be nice to use alphanumeric codes instead.
No behavior change expected.
Test plan:
[1] Run t/db_dependent/Upload.t
[2] Verify that a regular upload with tools/upload.pl still works.
[3] Rename upload_path in your koha-conf.xml. Restart Plack, flush the cache
and try to upload to a category. Correct error message?
[4] Upload the same file twice to the same category.
Correct error message the second time?
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Bug 19633: (QA follow-up) Really remove these ugly numbers
See BZ comment5. We now remove the numbers also from the constant names.
Test plan:
Read the changes.
Git grep "ERRCODE_"
Run t/db_dependent/Upload.t (Note: You may see one failure here; it is fixed
on bug 20727. So depends on who reaches master first.)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Trivial changes.
Adding C4::Context->temporary_directory to subsequent tmpdir call in Plugins.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
To test:
1 - Edit a framework and add the 'upload/pl' plugin to the 856 field
2 - Attach a pdf to one record
3 - Attach a different type of file to another
(To attach, edit a record in the framework edited above and find the
helper by the 856 field)
4 - Click the links on the record
5 - Verify the pdf opens inline
6 - Verify the other file does not
7 - prove t/db_dependent/Upload.t
Signed-off-by: Claire Gravely <claire.gravely@bsz-bw.de>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The test needs the code level in the error hash now.
And POD is adjusted accordingly.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
As noted on bug report 17669, the return values of delete (both singular
and plural), delete_missing and delete_temporary should be consistent.
Removed the if-construction around full_path. We do not need it; in the
very exceptional case that full_path would be empty, we should delete
the record since the file is missing.
Adjusted POD and unit tests accordingly.
Test plan:
Run t/db_dependent/Upload.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
If you deleted files from the upload directories manually, or you rebooted
with files in the temporary upload folder, or for some other reason have
records without a file, you may want to cleanup your records in the
uploaded_files table.
This patch adds the method delete_missing to Koha::UploadedFiles. It also
supports a keep_record parameter to do a dry run (count the missing files
first).
Also, we add an option --uploads-missing to cleanup_database. If you add
the flag 1 after this option, you will delete missing files. If you add the
flag 0 or only use the option, you will count missing files.
A subtest is added to Upload.t for delete_missing tests.
Test plan:
[1] Run t/db_dependent/Upload.t
[2] Upload a file and delete the file manually.
[3] Run cleanup_database.pl --uploads-missing
It should report at least one missing file now.
Check that the record has not been deleted.
[4] Run cleanup_database.pl --uploads-missing 1
It should report that it removed at least one file.
Check that the record is gone.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
See Bugzilla comment36 (QA request).
Koha::UploadedFile->delete
Returns 1, 0E0 or -1 (unknown).
Koha::UploadedFiles->delete and delete_temporary
Returns number of deleted records, 0E0 or -1.
POD lines are corrected accordingly. Unit tests adjusted too.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Added a note that Koha::Object->delete itself is not completely consistent
with Koha::Objects->delete (DBIx). The singular may return 1 when it
gets 0E0 from DBIx, while the plural one may return 0E0. But should be
handled somewhere else.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
As requested by QA on comment33.
If the pref is 0 or the overriding command line parameter is 0, all
temporary files will be deleted. But if the pref is NULL or empty string,
we will not delete files.
Also adjusted the description of the preference in this regard.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Requested by QA on comment25.
Result of:
git grep -l Upload_PurgeTemporaryFiles_Days | xargs sed -i -e "s/Upload_PurgeTemporaryFiles_Days/UploadPurgeTemporaryFilesDays/g"
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Test plan:
Run t/db_dependent/Upload.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch makes the following changes, as requested by QA:
[1] UploadedFile->delete always calls SUPER::delete.
The return value normally comes from SUPER::delete; if removing the
file failed, we return false. Two warns are kept.
Since delete does no longer return the filename, a few changes were
needed in tools/upload.pl.
[2] Method getCategories is moved to UploadedFiles. Script tools/upload.pl
now only contains one call. Added a use C4::Koha.
[3] Calls UploadedFiles->delete as class method. As a result I removed
method delete_errors for now; may be reconsidered on a new report.
[4] Adjusted three ->search calls for id and public to ->find calls.
[5] If you pass no id to upload.pl when deleting, you don't get an alert.
All by all, we got rid of 15 lines !
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch adds some documentation lines.
And mainly rearrangs the tests in Upload.t. The 'basic CRUD testing' is
not needed separately any more. A new test catches the "file missing"
warn.
Test plan:
[1] Run perldoc on UploadedFile[s].pm
[2] Run t/db_dependent/Upload.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Why? Koha::Uploader now only contains the actual CGI upload. The new name
better reflects its handler status.
Pragmatically, the difference between Uploaded and Uploader makes it
easier to specifically search for them in the codebase.
Test plan:
[1] Run t/db_dependent/Upload.t.
[2] Add an upload via the interface.
[3] Check the code:
git grep "Koha::Upload;"
git grep "Koha::Upload\->"
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Class method getCategories has no strict binding to Upload.pm. While
Upload.pm is now restricted to the actual uploading process with CGI
hook, this routine fits better in the UploadedFile package.
Class method httpheaders can be moved as well for the same reason. Note
that it actually is an instance method. The parameter $name is dropped.
Test plan:
[1] Run t/db_dependent/Upload.t.
[2] Check the categories in the combo box of tools/upload.
[3] Check a download via tools/upload and opac-retrieve-file.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
The get routine actually returns records from uploaded_files. It should be
possible to replace its calls by direct calls of Koha::UploadedFiles.
This patch is the crux of this patch set. It deals with all scripts that
use Koha::Upload.
In the process we do:
[1] Add a file_handle method to Koha::UploadedFile. This was previously
arranged via the fh parameter of get.
[2] Add a full_path method to UploadedFile. Previously returned in the
path hash key of get. (Name is replaced by filename.)
[3] Add a search_term method too (implementing get({ term => .. }).
This logic came from _lookup.
[4] Add a keep_file parameter to delete method. Only used in test now.
Test plan:
[1] Run t/db_dependent/Upload.t
[2] Go to Tools/Upload. Add an upload, download and delete.
[3] Add another public upload , search for it.
Use the hashvalue to download via opac with URL:
cgi-bin/koha/opac-retrieve-file.pl?id=[hashvalue]
[4] Go to Tools/Stage MARC for import. Import a marc file.
[5] Go to Tools/Upload local cover image. Import an image file.
Enable OPACLocalCoverImages to see result.
[6] Test uploading a offline circulation file:
Enable AllowOfflineCirculation, and create a koc file (plain text):
Line1: Version=1.0\tA=1\tB=2
Line2: 2016-11-23 16:00:00 345\treturn\t[barcode]
Note: Replace tabs and barcode. The number of tabs is essential!
Checkout the item with your barcode.
Go to Circulation/Offline circulation file upload.
Upload and click Apply directly.
Checkout again. Repeat Offline circulation file upload.
Now click Add to offline circulation queue.
[7] Connect the upload plugin to field 856$u.
Enable HTML5MediaEnabled.
Upload a webm file via the plugin. Click Choose to save the URL,
and put 'video/webm' into 856$q. Save the biblio record.
Check if you see the media tab with player on staff detail.
(See also: Bug 17673 about empty OPACBaseURL.)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Since delete is not part of the upload process, we will move it now
to Koha::UploadedFile[s].
Deleting the file will be done in UploadedFile.
The (multiple) delete method in UploadedFiles refers to the single delete.
Test plan:
[1] Run t/db_dependent/Upload.t
The warning ("but file was missing") in the last subtest is fine;
the file did not exist. Will be addressed in a follow-up.
[2] Search for uploads on Tools/Upload. Clone this tab (repeat search on
a new tab in your browser).
[3] Delete an existing upload on the first tab.
[4] Try to delete it again on the second tab. Error message?
[5] Bonus points:
Add an upload. Mark the file immutable with chattr +i. Try to delete
the file. You should see a "Could not be deleted"-message.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
In the next set of patches we will start using these new classes in
Koha::Upload, and scripts using it.
This is just the starting point of that migration.
Test plan:
[1] Run t/db_dependent/Upload.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
The insert into authorised_values failed because of the missing FK
Test plan:
prove t/db_dependent/Upload.t
should return green
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
At some point I saw some gain in warning about a missing config line.
But these warns are actually not needed here.
Removing them here.
Test plan:
Run t/db_dependent/Upload.t while having upload_path in your koha-conf and
also without it.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
As requested by QA, the sub should be moved.
Test plan:
See previous patches.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Brendan Gallagher <bredan@bywatersolutions.com>
The three permissions in tools/upload-file.pl are moved to a custom
routine in DBIx's UploadedFile.pm. An additional granular permission
upload_general_files is added. (The dbrev patch will contain it too.)
At some point in time this could be moved to a Koha::Object class.
The routine is tested in Upload.t.
Test plan:
[1] Run t/db_dependent/Upload.t
[2] If you only apply this patch, you can also test uploading with a
user that has edit_catalogue but does not have tools or circulate.
Upload status should say: Denied.
Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Brendan Gallagher <bredan@bywatersolutions.com>
To make life easier when multiple instances are uploading temporary
files to Koha, this patch adds the database name to the upload subfolder
in your /tmp folder.
Note: Although multiple instances could share the same subfolder for
temporary storage (hashvalue is based on a timestamp too), it will be
better to separate them for efficient housekeeping (removing older or
partial files with a cronjob etc.)
Since multiple instances come with separate permissions, keeping them in
separate folders will be much simpler.
Permanent storage is not affected by this patch. The location of permanent
storage is ruled by the upload_path in each config file. Sharing that
space is not recommended too.
Although it may not be strictly necessary yet to remove files from the old
temp storage folder (before the 3.22 release), the accompanying db rev
performs that housekeeping task.
Test plan:
[1] Do not yet apply this patch. Upload a temporary file (use the
tools/upload.pl script without selecting a category).
[2] Check /tmp/koha_upload.
[3] Apply this patch. Run the db rev with web installer.
[4] Upload another temporary file.
[5] Check /tmp for folder [your_database]_upload.
[6] Check that /tmp/koha_upload is gone and the associated records too.
[7] Run the adjusted t/db../Upload.t
Followed test plan. Could not do steps before applying patch [1]-[2]
(I suppose a local permission problem).
After applying patch and updating db file appears in /tmp/[database]
as expected.
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Brendan Gallagher brendan@bywatersolutions.com
This patch makes following requested changes:
[1] Renamed categorycode to uploadcategorycode (with atomicupdate file)
[2] Confirmation before deleting an uploaded file
[3] Label for Public checkbox can be clicked
[4] Back button on result list (not in plugin mode)
[5] Correct additional variable interpolation from sql queries
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Tested the db revision, unit test.
Uploaded temporary and permanent file, incl. plugin mode.
Deleted an upload.
Ran stage-marc-import.pl.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch introduces Koha::Upload. It will replace the modules
C4::UploadedFile.pm and the new C4::UploadedFiles.pm (from BZ 6874).
It also includes a new unit test.
NOTE: This unit test will replace the test for UploadedFiles.pm. It will no
longer use dependency Test::CGI::Multipart. We are now mocking CGI and its
hook to achieve the same result.
Test plan:
[1] Run t/db_dependent/Upload.t.
Note that if you see a WARNING, you will still need to add an entry
upload_path to your koha-conf.xml. Or you need to give write
permission to your Koha instance user for that folder.
Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
