Adrive.com, the Transfer Remote File Feature and server backup

I've had my basic Adrive.com account, with 50GB of free backup just sitting with no purpose for several years. I have had no reason to use it, mostly because the basic service only allows you to upload through a crappy little java applet and allows a max of 2ooo files at a time. For whatever reason, this did not cut it for me. Then one day, I gave adrive another try and ran across a link to Transfer Remote File, very nice! It works much the way you'd expect. Give it a file on the internet and it will download it and put it in the folder of your choosing on adrive.

There are stipulations of course, for instance, only one file at a time, no queuing mechanism, oh and no url's with question marks (?) in them. Damn.. no question marks? Adrive, have you any idea how many people are using server side scripting languages to deliver downloads these days? Anywho, I found out about the question mark issue a little late, way after I had devised use for adrive: as a server backup system. So, I had to adapt.

I have my own linode vps, just hanging out for projects and some light hosting. I also have some hefty folders full of files on that server, so I thought I would make a system for backing up folders using some encapsulation method. The constraints were that I would have to zip or tar the folders that I wanted, since the remote transfer only takes one at a time and no question marks in the url's. So I set to work. I was thinking of using a php script along with mod_rewrite for apache, so if I typed in

http://dl.server.com/Folder I Want.tar/zip

the php script would kick in and deliver Folder I Want as a tar/zip archive to adrive on the fly, which would be awesome.

Why not just make the archives first and allow them to be downloaded? There are a few reasons, the first and most important to me was storage. Putting 1-2GB of additional data onto disk temporarily is not an option, I'm strapped for space as it is. Secondly, is obscurity, having no discernible file structure to the outside world is nice and eliminates the need for the folders to be in the web root directory.

Implementing zip on the fly was my first task because it offered compression which saves on bandwidth. This failed miserably, however, because php offers no native zip streaming functions. There are a few libraries floating around out there, the most notable I found was ZipStream-PHP written by Pablotron, but when I looked at the documentation, its not suited for recursively adding folders and has a lot of overhead, creating the archive twice, once to calculate file size and the second to stream to the end user. Truth be told, I could probably make pablotrons library work if I wanted to spend any time on it, but I don't, so I won't - for now.

At this point, I'm just hoping that streaming a tar archive is easier, as it ought to be because there is no compression for tar archives. After some googling, I found a few php commands for streaming a tar archive. Here is my php script which I put in my webroot folder for dl.server.com

file.php

<?php

$dir = "/home/mike/torrent/";

$dirname = empty( $_GET['dir'] ) ? "hiphopapotomus" : $_GET['dir'];

// if there was no directory by that name then 404
if ( !file_exists( $dir . $dirname . "/" ) ){
header('HTTP/1.0 404 Not Found');
echo "<h1>404 File Not Found!</h1>";
echo "<p>The file that you have requested could not be found.</p>";
exit();
}

// remove script time limit to allow for download
set_time_limit(0);

$filename = $_GET['dir'];

// send browser file headers
header('Content-type: application/x-tar');
header('Content-Disposition: attachment; filename="' . $filename . '.tar');

// add some escape characters
$filename = escapeshellarg($filename);

// the C argument is so that doesn't get included in your tarball extraction
$cmd = "tar cC $dir $filename";

// stream tar file
$fh = popen($cmd, 'r');

while (!feof($fh)) {
print fread($fh, 8192);
}

pclose($fh);

?>



MOD_REWRITE

Now that we have the script running properly, get mod rewrite working so that Adrive doesn't bitch about question marks. Installing and setting up mod_rewrite for apache is beyond the scope of this post, however, I'm sure you're a resourceful person and can use google.
As it stands now, a file request looks like this:

http://dl.server.com/file.php?dir=Folder I Want

We need to make it

http://dl.server.com/Folder I Want.tar

so, with a little .htaccess magic we can make that happen.
Create a new .htaccess file in the webroot folder

Options +FollowSymlinks -Indexes
RewriteEngine on
RewriteRule ^(.*)\.tar$ file.php?dir=$1 [NC]

So, I bet you want to know what this does. The magic is in the RewriteRule line.

PART 1

RewriteRule ^(.*)\.tar$

• ^ means beginning of expression - meaning stuff after http://dl.server.com/ - in this case: Folder I Want.tar

• () parentheses will denote that we want to store the thing that we match inside as a variable
• . - means that we want to match one character - in this case, the first character is 'F'
• * - means that we want to match 0 or more of the thing before it.

.* will match:

Folder,
Folder I
Folder I Want
Folder I Want.tar,
or any permutation you can think of

• \. - since the symbol '.' means something special, if we actually want to find a period, we have to tell apache that we just want the period, and that means using the escape character '\' so \. means -> find a period

so far:

^(.*)\. will match

Folder I Want.

where

Folder I Want

will be stored in a variable that we can use later.

now for the magic

• tar$ will match anything ending in tar, $ means end, so it will match 'Folder I Want.tar'

The full regular expression(regex) ^(.*)\.tar$ will match:

anythingyoucanconcieveof.tar

In plain english it says match anything that starts with something and ends with .tar

PART 2

file.php?dir=$1 [NC]

The second part of this line tells apache what to convert this to if it finds the expression

• file.php - is the name of the php script we want to handle this business.
• ? - ? means that everything after this is data from the requester
• dir - is the name of the php variable $_GET['dir'] in our script
• = - is setting $_GET['dir'] equal to anything that comes after it
• $1 - remember in the first part when 'Folder I Want' is saved to a variable? Well $1 is that variable
• [NC] - means no case, equivalent to case insensitive so Folder I Want.TaR would match

What does all this do?

http://dl.server.com/Folder I Want.tar
Becomes
http://dl.server.com/file.php?dir=Folder I Want

All of this happens in the background on the server side, so Adrive never notices any of this happening. Mod rewrite changes the requested url to what you want and then performs the actual http request, so the file.php script is called whenever Something.tar put in the url. Nice huh?

NOTES:

• You can request folder names with space in them by just tying them exactly as they appear EX: Folder Name.tar
• Adrive will complain that it does not know the size of the file being downloaded. This is because we never told adrive how large it was when we began the stream. This is an inherent problem with the file.php script, which I will someday attempt to rectify.

Troubleshooting:

I ran into a problem with mod_rewrite not working correctly (at all actually). This problem stemmed from the httpd.conf file for apache, well, actually from the vhost file in the /etc/apache2/sites-available/ (im running Ubuntu server)

My vhost file had the following line in my directory declarations twice:

AllowOverride None

I changed them to

AllowOverride All

Ran apache2ctl graceful

Viola! Worked. Now, if you're using shared hosting, you probably wont be able to change these files, but then again, I would think your hosting provider would allow overrides with .htaccess files from the start.

This concludes my Adrive mania. Oh, the steps we take to not pay money for a backup solution! I know it is unnecessarily long, but I am conditioned to adhere to mathematical rigor, even when no math is involved.

Bill Williams Fractal Indicator for Think or Swim

Here's a continuation of the Bill Williams indicator set, the Fractal indicator coded in thinkscript.

When setting this up with TOS, you can change the type of object that appears on the chart. I have found some issue using the up and down arrows, so I plot colored dots. I have not set any default objects in this code, because there is no ideal object to plot on the chart.

Fractal Indicator:

# Bill Williams fractal indicator
# written by Mike Lapping
#
# Make sure that you change the settings for this indicator so that it plots arrows
# up for upfractal plots
# down for downfractal plots
#
# can be used and modified by anyone for any reason. do not sell.

def isupfractal;
def isdownfractal;

# Looking for high and low series of equalities
# checking for possible fractal formation

rec hicount = if (high == high[1], hicount[1] + 1, 0);
# if the last bar high is the same as this bar's high, increment
# otherwise set false(0)

rec hivalid = if ((hicount[1] == 0 and hicount == 1 and high > high[2] and high > high[3]) or (hicount[1] and hicount and hivalid[1] )
or (hicount[2] and hivalid[2] and high == high[2] and high > high[1]), 1, 0) ;
# set hivalid to true(1)
# if we are entering an equality series, check if the 2 bars preceding the first equal bar # are lower than the current bar
# or if the last bar was an equal bar and this bar is an equal bar and the current equal
# series is valid
# or if we skipped over a lower bar but two bars ago we had an equal bar and that was a
# valid fractal
# otherwise it is false

rec locount = if (low == low[1], locount[1] + 1, 0);
rec lovalid = if ((locount[1] == 0 and locount == 1 and low < low[2] and low < low[3])
or (locount[1] and locount and lovalid[1] )
or (locount[2] and lovalid[2] and low == low[2] and low < low[1]), 1, 0) ;



# Checking for a traditional or non-standard up fractal

isupfractal = if(((hicount and hivalid) or (high > high[1] and high > high[2])) and high > high[-1] and high > high[-2], high, 0);
# Ok this is complicated, basically its checking if there were a series of equal bars and
# if the two bars before the first equal bar were lower
# or if the last 2 bars were lower than the current bar
# and if the two following 2 bars are lower than the current bar

# Checking for a traditional or non-standard down fractal

isdownfractal = if(((locount and lovalid) or (low < low[1] and low < low[2])) and low < low[-1] and low < low[-2], low, 0);

plot upfractal = if( isupfractal, isupfractal + (1 * tickSize()), double.nan);

plot downfractal = if( isdownfractal, isdownfractal - (1 * tickSize()), double.nan);

# This business with the tickSize() function is basically putting the arrow further away fro
# from the bar so that the fractal is easier to read

Bill Williams Alligator for Think or Swim

Coming from the forex world, Metatrader is well-equipped to handle trading based on Bill Williams understanding of chaos theory and fractal analysis, however Think or Swim is not. I have scoured the web for suitable indicators, but in the end, I had to write my own in thinkscript.

What Think or Swim Has:
Accelerator Oscillator - for the purists who enjoy Bill's older books
Awesome Oscillator - the 5,34 macd where you can count the elliot wave, etc

What TOS lacks:
The Alligator on the chart
Fractal Indicator on the chart


The Alligator

####
# Bill Williams Alligator for Think or Swim
# Mike Lapping 2010

# Extra steps were taken because I cannot find proper
# documentation of the average() function
# can be used and modified by anyone for any reason. do not sell.
#
####

## we will now work on the SSMA called jaws

def JOffset = 8;
def Javg13 = ((high[12 + JOffset] + low[12 + JOffset]) / 2);
def Javg12 = ((high[11 + JOffset] + low[11 + JOffset]) / 2);
def Javg11 = ((high[10 + JOffset] + low[10 + JOffset]) / 2);
def Javg10 = ((high[9 + JOffset] + low[9 + JOffset]) / 2);
def Javg9 = ((high[8 + JOffset] + low[8 + JOffset]) / 2);
def Javg8 = ((high[7 + JOffset] + low[7 + JOffset]) / 2);
def Javg7 = ((high[6 + JOffset] + low[6 + JOffset]) / 2);
def Javg6 = ((high[5 + JOffset] + low[5 + JOffset]) / 2);
def Javg5 = ((high[4 + JOffset] + low[4 + JOffset]) / 2);
def Javg4 = ((high[3 + JOffset] + low[3 + JOffset]) / 2);
def Javg3 = ((high[2 + JOffset] + low[2 + JOffset]) / 2);
def Javg2 = ((high[1 + JOffset] + low[1 + JOffset]) / 2);
def Javg1 = ((high[0 + JOffset] + low[0 + JOffset]) / 2);

def SSMAJaws = ( Javg1 + Javg2 + Javg3 + Javg4 + Javg5 + Javg6 + Javg7 + Javg8 + Javg9 + Javg10 + Javg11 + Javg12 + Javg13) / 13;

Plot Jaws = SSMAJaws;

######### Now working on Lips
def LOffset = 5;
def Lavg8 = ((high[7 + LOffset] + low[7 + LOffset]) / 2);
def Lavg7 = ((high[6 + LOffset] + low[6 + LOffset]) / 2);
def Lavg6 = ((high[5 + LOffset] + low[5 + LOffset]) / 2);
def Lavg5 = ((high[4 + LOffset] + low[4 + LOffset]) / 2);
def Lavg4 = ((high[3 + LOffset] + low[3 + LOffset]) / 2);
def Lavg3 = ((high[2 + LOffset] + low[2 + LOffset]) / 2);
def Lavg2 = ((high[1 + LOffset] + low[1 + LOffset]) / 2);
def Lavg1 = ((high[0 + LOffset] + low[0 + LOffset]) / 2);

def SSMALips = (Lavg1 + Lavg2 + Lavg3 + Lavg4 + Lavg5 + Lavg6 + Lavg7 + Lavg8) / 8;

Plot Lips = SSMALips;

######## Work on teeth
def TOffset = 3;

def Tavg5 = ((high[4 + TOffset] + low[4 + TOffset]) / 2);
def Tavg4 = ((high[3 + TOffset] + low[3 + TOffset]) / 2);
def Tavg3 = ((high[2 + TOffset] + low[2 + TOffset]) / 2);
def Tavg2 = ((high[1 + TOffset] + low[1 + TOffset]) / 2);
def Tavg1 = ((high[0 + TOffset] + low[0 + TOffset]) / 2);

def SSMATeeth = (Tavg1 + Tavg2 + Tavg3 + Tavg4 + Tavg5) / 5;

Plot Teeth = SSMATeeth;

## Set Colors for the balance lines

Jaws.SetDefaultColor(Color.BLUE);
Lips.SetDefaultColor(Color.UPTICK);
Teeth.SetDefaultColor(Color.DOWNTICK);

Remember to time shift the chart at least 9 bars so that you can have the full effect of the alligator. This is done by:

Right click on chart > Style > Settings.

In the new window, go to the "Scale and Axis" tab, half way down, change the value for "Expansion" to 8 or more. The Jaws are projected 8 bars into the future.

Hope everyone benefits from this!

-Mike