A quick background why I wanted this: I wanted to use an editor for writing blog posts that better supported tags.  I’m really not that facile at typing the < and > characters for HTML tags.  Also, I prefer my nascent post to grow somewhere I can get to it wherever I reasonably might get an idea.

I could use Subversion, Unison, or similar synchronization software. It’s discomfiting to use version control software for non-versioning purposes, but the main problem I have is remembering to start the synchronization.  For me, remembering to save the file was quite an accomplishment already!  :)

I recalled WebDAV had something to do with shared web drives (WebDrive also came to mind).  As it turns out, Apache (I’m using 2.0 on Debian) has support for WebDAV as a module; I conveniently found instructions to set up WebDAV on Debian.  I then found out that Novell produces an excellent WebDAV client for Windows, NetDrive (I don’t know why Novell doesn’t provide it).

I ran into frustrating problems for awhile, and eventually discovered that it was because the instructions I used to set up WebDAV recommend using “Digest Authentication” to control access to the WebDAV location.  NetDrive doesn’t know how to work with this.  Switching Apache to use “Basic Authentication” (and remembering to use htpasswd2 instead of htdigest to generate the user/password pairs) solved that problem. I also went a little further and exposed the WebDAV URL via HTTPS only.

So far, everything works beautifully.  My WebDAV folder shows up as a drive letter on my Windows boxes.  If I had a Linux desktop, I’d have even more choices of WebDAV clients.  If I’m in a pinch (e.g. foreign computer), I can still SSH to my server and edit files locally.  It’s just me, so I don’t have to worry about conflicts and similar muti-user issues.

FYI, I eventually settled on NVu as my HTML editor;  NVu is derived from Netscape Composer.  WYSIWYG HTML editors aren’t great, but I’ll give up optimal HTML if I can concentrate on content rather than syntax.

Subversion support in Ant, while present, isn’t that great.  It handles the basics, but not much else.   One are of improvement that I had needed was in accessing the repository metadata, specifically the revision number that was being used to build, so that the distributable could embed that as part of the filename.  Previously, I had been using the build date, but that’s obviously not accurate.

To support cross-platform, I ended up with JavaSVN from Tmate.  It didn’t totally do what I needed to do, but at least I could call it as a command line application to recursively display the information about every file in the repository.  After that, it’s only a matter of grep‘ing for “Revision:” and finding the highest revision.  I also had to write a simple sorting class to extend Ant (which has tail and head but no sort).  It works well, in a wasteful sort of way.

Here’s what I did:
<target name="checkRevision">
    <property name=”source-root” value=”.”/>
    <java classname=”org.tmatesoft.svn.cli.SVN”
       dir=”${source-root}” fork=”true” outputproperty=”revision.list”>
      <arg line=”info -R .”/>
      <classpath>
        <pathelement location=”${tools}/svnant/javasvn.jar” />
        <pathelement location=”${tools}/svnant/javasvn-cli.jar” />
      </classpath>
      <redirector>
        <outputfilterchain>
          <linecontainsregexp>
            <regexp pattern=”^Revision:”/>
          </linecontainsregexp>
          <deletecharacters chars=”Revision: “/>
        </outputfilterchain>
      </redirector>
    </java>

    <echo message=”Latest commited revision: ${coconut.revision}”/>
  </target>


Not that this is what’s kept me busy in technology for the last couple months, but it’s easy to post about as I try to recover from the holidays’ hiatus.

Tiffany’s (that high-end jewelry store) has many picture of its products on its site. The photography is quite good, and most of the pictures have large, high quality versions; makes for good desktop wallpaper. However, it’s tedious to go to each link and save each picture individually - there’s a lof of them.

Regular mirroring doesn’t work too well. The site uses a lot of Javascript, in particular to pop up windows with the englarged picture inside. Fortunately, most of Tiffany’s zoomed-in pictures of its products on the website have a regular format, e.g. http://www.tiffany.com/images/products/zoom_images/12259417_xl.jpg. Tather than try to be “smart” and figure a way to decipher correct links from the website programmatically, I chose to be “dumb” and do a brute force search on every possible number.

This saves me a lot of brain time, at the expense of computer time. Fortunately, the computer doesn’t value its time as much as I value mine, so it’s happy to run for a few days. It takes quite awhile to test one hundred million possibilities. The script below isn’t my first attempt at trying to get them all, but it’s certainly the fastest. I ran it on a Linux box, and it’s mainly curl that makes it possible.

Some of the optimizations I made along the way:

• A naive HTTP download for each possibility is quite slow, not so much from creating a new process each time, but from the network connection overhead and I/O.
• curl is much better than wget for these kinds of things, as one can specify a pattern and range of numbers to use, and it will also reuse the same HTTP connection. Truly, “a Client that groks the URLs”.
• It was faster to download the HTTP header and look at the advertised file type (e.g. image/jpeg) using grep than make a full HTTP request and examine the resultant downloaded data to determine the file type (e.g. with file). More so than I would have thought if the speedup was only due to less bytes over the wire.

The script follows:

 #!/bin/sh  # e.g. http://www.tiffany.com/images/products/zoom_images/12259417_xl.jpg START_ID=10000000  CLEAN_LOCK=clean.lock CURL_LOCK1=curl1.lock CURL_LOCK2=curl2.lock CURL_LOCK3=curl3.lock CURL_LOCK4=curl4.lock  function remove_non_jpegs {   # Delete the non-JPEG files   START_ID=$1   END_ID=$2   CLEAN_LOCK=$3   echo "[`date +'%x %X'`]" Cleaning $START_ID-$END_ID    ID=$START_ID   while [ $ID -le $END_ID ];   do     JPG=${ID}_xl.jpg     if [ -z "`grep -m 1 'image/jpeg' $JPG`" ]; then       rm $JPG     else       echo $JPG is O.K.       rm $JPG       wget -q "http://www.tiffany.com/images/products/zoom_images/$JPG"     fi     ID=`expr $ID + 1`   done    rm -f $CLEAN_LOCK    echo "[`date +'%x %X'`]" Done cleaning $START_ID-$END_ID }  function slurp_data {   curl -I -s "http://www.tiffany.com/images/products/zoom_images/[$1-$2]_xl.jpg" -o "#1_xl.jpg"   rm -f $3 }  while [ $START_ID -lt 20000000 ]; do   END_ID=`expr $START_ID + 4000`    INTERVAL=`expr $END_ID - $START_ID`   INTERVAL=`expr $INTERVAL / 4`    ONE_QUARTER_ID=`expr $START_ID + $INTERVAL`   TWO_QUARTER_ID=`expr $ONE_QUARTER_ID + $INTERVAL`   THREE_QUARTER_ID=`expr $TWO_QUARTER_ID + $INTERVAL`    echo "[`date +'%x %X'`]" Getting $START_ID-$END_ID   lockfile $CURL_LOCK1 $CURL_LOCK2 $CURL_LOCK3 $CURL_LOCK4   slurp_data $START_ID $ONE_QUARTER_ID $CURL_LOCK1 &   slurp_data `expr $ONE_QUARTER_ID + 1` $TWO_QUARTER_ID $CURL_LOCK2 &   slurp_data `expr $TWO_QUARTER_ID + 1` $THREE_QUARTER_ID $CURL_LOCK3 &   slurp_data `expr $THREE_QUARTER_ID + 1` $END_ID $CURL_LOCK4 &   lockfile $CURL_LOCK1 $CURL_LOCK2 $CURL_LOCK3 $CURL_LOCK4   rm -f $CURL_LOCK1 $CURL_LOCK2 $CURL_LOCK3 $CURL_LOCK4   echo "[`date +'%x %X'`]" Done getting $START_ID-$END_ID    lockfile $CLEAN_LOCK   remove_non_jpegs $START_ID $END_ID $CLEAN_LOCK &    START_ID=`expr $END_ID + 1` done