Blog => Blob => Jello

Random Stuff.

Mass uploading files to ownCloud (the painful way)

icon 2014-10-02 - No comments

So, I've managed to install ownCloud on my Centos with lighttpd.

But now I've got tons of files to push. Like loads of files. With weird file names and everything. OK, maybe not so weird, but anyway, let's see how we can do that.

First, find a comand line utility

Obviously, the first thing, one should consider should be the mirall ownCloud desktop client GUI, It's pretty useful if you have a whole desktop, and GUIs everywhere and if you want to keep the files in sync and all. But right now, all I want to do is a single one shot sync from a headless server, and there a generic ways to upload those files, since ownCloud has a WebDAV interface.

So let's search for webdav clients : 
$ sudo yum search webdav | grep client
cadaver.x86_64 : Command-line WebDAV client
neon.x86_64 : An HTTP and WebDAV client library
Something tells me cadaver might be the easiest solution for me, here. Let's go !
yum install cadaver
For the record, cadaver is based on the neon library.

Cadaver uses the ~/.netrc file to store credentials and avoid having to enter the webDAV username & password every 20 seconds. Its format is explained in the man cadaver. (For your own sake, I strongly suggest you do NOT google man cadaver). But anyway, just put those 3 lines in the file.
machine your.owncloud.hostname
  username your_username
  password your_password
If put correctly, this should now work out of the box:
$ cadaver https://your.owncloud.hostname/owncloud/remote.php/webdav
dav:/owncloud/remote.php/webdav/> ls
Listing collection `/owncloud/remote.php/webdav/': collection is empty.
Well, I have not uploaded anything to my owncloud yet. Let's try to upload something :
dav:/owncloud/remote.php/webdav/> put bash-argsparse-1.6.tar.gz 
Uploading bash-argsparse-1.6.tar.gz to `/owncloud/remote.php/webdav/bash-argsparse-1.6.tar.gz':
Progress: [=============================>] 100.0% of 44822 bytes succeeded.
dav:/owncloud/remote.php/webdav/> ls
Listing collection `/owncloud/remote.php/webdav/': succeeded.
        bash-argsparse-1.6.tar.gz          44822  Oct  1 13:12
C00l.

OK, but put only works for a single file. To upload many files, cadaver has an mput command, but it has 2 annoying limitations:
  1. you cannot specify the destination directory (webDAV talks about collections, but nevermind that) on the webDAV server
  2. it is not recursive -_-
The first limitation can easily be worked around by performing a cd before the mput. The second one is less trivial to work around be easy enough, if you ask me.

So, considering we want to recursively upload a directory named pics in the root collection of the webDAV server, proceed in two step :
  • first create all the collections,
  • then upload all the files.

If you don't want to end up uploading your files in random directories, then you will want to do it in 2 steps. You see, cadaver offers no easy way to control a command has been succesfully performed (or not). So manual check the result after the first step then perform the second one.

Create all the collections

So, first thing first, to create all the collections on the server, using bash4, let's enter the directory where the pics directory is and run:
shopt -s globstar
dirs=( pics/**/ )
dirs=( "${dirs[@]// /\\ }" )
cadaver https://your.owncloud.hostname/owncloud/remote.php/webdav <<<"mkcol ${dirs[*]}"
It should produce an output like:
dav:/owncloud/remote.php/webdav/> mkcol pics/ pics/Japan/ pics/Japan/After\ Kiyomizu/ pics/Japan/Akiba/ pics/Japan/Ghibli/ pics/Japan/Hakone/ pics/Japan/Heian/ pics/Japan/Hello\ Kitty/ ...
Creating `pics/': succeeded.
Creating `pics/Japan/': succeeded.
Creating `pics/Japan/After Kiyomizu/': succeeded.
Creating `pics/Japan/Akiba/': succeeded.
Creating `pics/Japan/Ghibli/': succeeded.
Creating `pics/Japan/Hakone/': succeeded.
Creating `pics/Japan/Heian/': succeeded.
Creating `pics/Japan/Hello Kitty/': succeeded.
...
dav:/owncloud/remote.php/webdav/>
Connection to `your.owncloud.hostname' closed.
So just check you have everything you need either on the ownCloud web interface, or navigating through the webDAV URL using cadaver command line.

As you might have noticed the only character that has been escaped is the space char. If your directories or files have more funny chars cadaver does not understand as part of the name, just try to escape them with the same method. Avoid external commands if you can.

Now about uploading files

That is s a bit trickier, since regular files needs to be to filtered directories. But using the get_files function below, it gets a bit easier.

After defining the get_files function, both user and bash users can do this to upload the files:
get_files() {
    files=( * )
    local i
    for i in "${!files[@]}"
    do
        [[ ! -f ${files[$i]} ]] && unset "files[$i]"
    done
}
shopt -s globstar
for dir in pics/**/
do
  (
    cd "$dir" || exit
    get_files
    files=( "${files[@]// /\\ }" )
    cadaver https://your.owncloud.hostname/owncloud/remote.php/webdav <<EOF
cd ${dir// /\\ }
mput ${files[*]}
EOF
  )
done

( Of course, zsh users can just use this instead of the get_files function:
files=( *(.) )
)

And voila.

And don't forget to clean your mess after the upload and remove the file that contains your ownCloud account password. :-P
rm -v ~/.netrc

icon Tags de l'article : , , ,