In my previous post, I've described a kinda complicated but command-line based user-level way to mass upload files to ownCloud, using the cadaver WebDAV client.
Well. Erm... There are WebDAV filesystems implementations out there. See ?
$ yum search webdav|grep -i filesystem
davfs2.armv6hl: A filesystem driver for WebDAV
So I did yum install davfs2
-ed, and that just worked. And I've configured davfs2, following a tutorial like these one is fine enough:
Note that disabling locks is indeed required for ownCloud, so I needed to edit davfs2.conf
use_locks 0
After a mount /mnt/owncloud
to locally bind your ownCloud webDAV to a directory, to make recursively upload a full directory tree, it is easy as this:
rsync -vr --progress --bwlimit=123 /path/to/my/directory/. /mnt/owncloud/my_sub_dir/.
In case the rsync
stops in the middle of the upload, the command line can be entered again to resume the upload.
So, no cadaver
, no shell
dark magic, just a mount
and an rsync
.
Easy.
Tags de l'article : Linux, WebDAV, davfs, ownCloud
So, I've managed to install ownCloud on my Centos with lighttpd.
But now I've got tons of files to push. Like loads of files. With weird file names and everything. OK, maybe not so weird, but anyway, let's see how we can do that.
First, find a comand line utility
Obviously, the first thing, one should consider should be the mirall ownCloud desktop client GUI, It's pretty useful if you have a whole desktop, and GUIs everywhere and if you want to keep the files in sync and all. But right now, all I want to do is a single one shot sync from a headless server, and there a generic ways to upload those files, since ownCloud has a WebDAV interface.
So let's search for webdav clients :
$ sudo yum search webdav | grep client
cadaver.x86_64 : Command-line WebDAV client
neon.x86_64 : An HTTP and WebDAV client library
Something tells me cadaver might be the easiest solution for me, here. Let's go !
yum install cadaver
For the record, cadaver is based on the neon library.
Cadaver uses the ~/.netrc
file to store credentials and avoid having to enter the webDAV username & password every 20 seconds. Its format is explained in the man cadaver
. (For your own sake, I strongly suggest you do NOT google man cadaver
). But anyway, just put those 3 lines in the file.
machine your.owncloud.hostname
username your_username
password your_password
If put correctly, this should now work out of the box:
$ cadaver https://your.owncloud.hostname/owncloud/remote.php/webdav
dav:/owncloud/remote.php/webdav/> ls
Listing collection `/owncloud/remote.php/webdav/': collection is empty.
Well, I have not uploaded anything to my owncloud yet. Let's try to upload something :
dav:/owncloud/remote.php/webdav/> put bash-argsparse-1.6.tar.gz
Uploading bash-argsparse-1.6.tar.gz to `/owncloud/remote.php/webdav/bash-argsparse-1.6.tar.gz':
Progress: [=============================>] 100.0% of 44822 bytes succeeded.
dav:/owncloud/remote.php/webdav/> ls
Listing collection `/owncloud/remote.php/webdav/': succeeded.
bash-argsparse-1.6.tar.gz 44822 Oct 1 13:12
C00l.
OK, but put
only works for a single file. To upload many files, cadaver has an mput
command, but it has 2 annoying limitations:
- you cannot specify the destination directory (webDAV talks about collections, but nevermind that) on the webDAV server
- it is not recursive -_-
The first limitation can easily be worked around by performing a cd
before the mput
. The second one is less trivial to work around be easy enough, if you ask me.
So, considering we want to recursively upload a directory named pics
in the root collection of the webDAV server, proceed in two step :
- first create all the collections,
- then upload all the files.
If you don't want to end up uploading your files in random directories, then you will want to do it in 2 steps. You see, cadaver offers no easy way to control a command has been succesfully performed (or not). So manual check the result after the first step then perform the second one.
Create all the collections
So, first thing first, to create all the collections on the server, using bash4, let's enter the directory where the pics
directory is and run:
shopt -s globstar
dirs=( pics/**/ )
dirs=( "${dirs[@]// /\\ }" )
cadaver https://your.owncloud.hostname/owncloud/remote.php/webdav <<<"mkcol ${dirs[*]}"
It should produce an output like:
dav:/owncloud/remote.php/webdav/> mkcol pics/ pics/Japan/ pics/Japan/After\ Kiyomizu/ pics/Japan/Akiba/ pics/Japan/Ghibli/ pics/Japan/Hakone/ pics/Japan/Heian/ pics/Japan/Hello\ Kitty/ ...
Creating `pics/': succeeded.
Creating `pics/Japan/': succeeded.
Creating `pics/Japan/After Kiyomizu/': succeeded.
Creating `pics/Japan/Akiba/': succeeded.
Creating `pics/Japan/Ghibli/': succeeded.
Creating `pics/Japan/Hakone/': succeeded.
Creating `pics/Japan/Heian/': succeeded.
Creating `pics/Japan/Hello Kitty/': succeeded.
...
dav:/owncloud/remote.php/webdav/>
Connection to `your.owncloud.hostname' closed.
So just check you have everything you need either on the ownCloud web interface, or navigating through the webDAV URL using cadaver command line.
As you might have noticed the only character that has been escaped is the space char. If your directories or files have more funny chars cadaver does not understand as part of the name, just try to escape them with the same method. Avoid external commands if you can.
Now about uploading files
That is s a bit trickier, since regular files needs to be to filtered directories. But using the get_files
function below, it gets a bit easier.
After defining the get_files function, both user and bash users can do this to upload the files:
get_files() {
files=( * )
local i
for i in "${!files[@]}"
do
[[ ! -f ${files[$i]} ]] && unset "files[$i]"
done
}
shopt -s globstar
for dir in pics/**/
do
(
cd "$dir" || exit
get_files
files=( "${files[@]// /\\ }" )
cadaver https://your.owncloud.hostname/owncloud/remote.php/webdav <<EOF
cd ${dir// /\\ }
mput ${files[*]}
EOF
)
done
( Of course, zsh users can just use this instead of the get_files
function:
files=( *(.) )
)
And voila.
And don't forget to clean your mess after the upload and remove the file that contains your ownCloud account password. :-P
rm -v ~/.netrc
Tags de l'article : Linux, WebDAV, cadaver, ownCloud
2014-10-09 - No comments