|
|
| View previous topic :: View next topic |
| Author |
Message |
Kosh
Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile
|
 Posted: Thu Feb 12, 2009 11:29 pm Post subject: Uploading (compressed) data vie shell script |
 |
|
I am guessing that the audience for a shell script to upload Census results is probably limited, particularly if your OS doesn't have a sufficiently UNIX-like shell environment, but I'll post what i have so far, anyway. 
Take the following code and save it as "censusup.sh" (or any name of your choosing) in a convenient directory that you have full permissions to. I put it in my normal "downloads" directory, because I already have a symlink to my CensusPlus.lua file there, but someplace like your home directory would also be fine. Note that you should not put it somewhere in the WoW directory tree. Oh yes, don't forget to make the script executable (chmod +x censusup.sh) 
| Code: | #!/bin/sh
myNAME=`basename $0`
cpNAME=**your username**
cpPW='**your password**'
echo "$myNAME: Checking for new version of CencusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ CensusPlus.lua -nt CensusPlus.lua.gz ] );
then {
echo " lua file newer -- recompressing...";
gzip -c -9 CensusPlus.lua > CensusPlus.lua.gz;
lastERR=$?
if ( [ $lastERR -ne 0 ] );
then {
echo "$myNAME: Error from gzip: $lastERR"
};
else {
echo " ...compressed. Starting upload..."
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
curl -F CensusPlus=@CensusPlus.lua.gz -F user=$cpNAME -F pw=$cpPW -w "\ncurl: transfered %{size_upload} bytes in %{time_total} seconds (%{speed_upload} b/s)\n" http://www.warcraftrealms.com/uniupload.php
lastERR=$?
echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
if ( [ $lastERR -ne 0 ] );
then {
echo "$myNAME: Error from curl: $lastERR"
};
else {
echo " upload successful."
};
fi
};
fi
};
else {
echo " compressed file up to date -- doing nothing.";
};
fi
cd $holdPWD
#eof |
(Note that the "curl" line is one long line.)
You will need to put your warcraftrealms.com username and password in the third and fourth lines. Keep the single quotes around the password, if you have any "special" characters in it.
Note that the script creates the gzip-compressed copy of the CensusPlus.lua file in the same directory as the script. It also assumes that the lua file (or a link to it) is also in the same directory. You can change this by hard-coding the path to the file in line 12, but I prefer having the symbolic link in the directory for convenience.
Note that I haven't spent a lot of time on this, so the error handling is rather simplistic (it just stops on errors), and it's fairly verbose. I also haven't gotten around to attempting to parse the response from the php script to look for logical errors. It just dumping out everything that's returned, so you need to look over the response to make sure that the "your name has been credited" line is there, and that no errors are reported. |
|
| Back to top |
|
 |
Rollie
Site Admin
Joined: 28 Nov 2004
Posts: 5374
Location: Austin, TX
WR Updates: 480,131
Rollie WR Profile
|
| Posted: Fri Feb 13, 2009 9:22 am Post subject: |
|
|
| well isn't that spiffy =) |
|
| Back to top |
|
|
Kosh
Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile
|
| Posted: Mon Feb 16, 2009 10:44 pm Post subject: |
|
|
Yep, I think so
A bonus to me is that I don't have to worry about hitting my head on the size limit (at least for quite a while), since I'm sending it compressed. |
|
| Back to top |
|
|
Kosh
Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile
|
| Posted: Wed Mar 04, 2009 1:12 am Post subject: |
|
|
Well, I put a little more work into my script. In addition to some cleanup, it now remembers if the compress succeeded but the upload failed, so a rerun of the script will retry the upload even if the lua file hasn't changed. It also strips down the server output (via awk) and also detects "logical" errors reported by the server-side script.
BTW Rollie, do any of the possible errors from uniupload.php start out with something other than "Error"? If so, then I will need to tweak the awk script.
Anyway, here's the updated version of censusup.sh:
| Code: | #!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME: Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ $cpFN -nt $cpFNgz ] ); then
{
echo " lua file newer -- recompressing...";
gzip -c -9 $cpFN > $cpFNgz;
lastERR=$?;
if ( [ $lastERR -ne 0 ] ); then
{
echo "$myNAME: Error from gzip: $lastERR";
needUPLOAD=-1;
};
else
needUPLOAD=1;
fi
};
else
{
# need to force another upload?
if ( [ "$1" == "-f" -o -e $myREUPFLAG ] ); then
{
echo " compressed file up to date, but upload needed";
needUPLOAD=1;
};
else
{
echo " compressed file up to date -- doing nothing.";
needUPLOAD=0;
};
fi
};
fi
if ( [ $needUPLOAD -gt 0 ] ); then
{
echo " Starting upload..."
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
-w "\n!!!curl %{size_upload} %{time_total} %{speed_upload}" \
http://www.warcraftrealms.com/uniupload.php \
| awk \
'/^$/ { next }
/^Checking your/
/^Found / { if( foundLine != "" ) foundLine = foundLine "; ";
if( $NF == "<=" )
foundLine = foundLine $(NF-2) " " $(NF-1);
else
foundLine = foundLine substr( $0, 7 );
next
}
/^Attempting / { if( foundLine != "" )
{ print "Found: " foundLine;
foundLine = "";
}
print $0;
next
}
/^Username/
/ will be credited /
/^!!!curl/ { kb = $2 / 1024;
print "\ncurl: transferred " kb "kb in " $3 " seconds (" kb / $3 " kb/s)";
}
/^Error/ { print "***\n" $0; exit 10 }
'
# copy the array of pipe statuses - $lastERR refs [0] (curl stat)
lastERR=( "${PIPESTATUS[@]}" )
awkERR=${lastERR[1]} # (status of awk)
echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
if ( [ $lastERR -ne 0 ] ); then
{
echo "$myNAME: Error from curl: $lastERR"
# flag the failed upload, so we can retry later
touch $myREUPFLAG
};
else
{
if ( [ $awkERR -ne 0 ] ); then
{
echo "$myNAME: Error from server during upload"
# flag the failed upload, so we can retry later
touch $myREUPFLAG
};
else
{
date "+ upload completed at %H:%M:%S"
rm -f $myREUPFLAG
};
fi
};
fi
};
fi
cd $holdPWD
#eof
|
The script woudl be a bit cleaner if I put the awk script in it's own file, but then another file would have to be floating around. Note that the 22 lines after the fourth line of the "curl" command is one long string constant (the awk command script).
In case you're wondering, here's an example of what the awk-massaged output looks like: | Code: | Checking your file
Found: CensusPlus Database; CensusPlus Info Data; Version: 4.2.2; Locale: US
Attempting to make entry
Username : Kosh<=
Kosh will be credited with this submission!
curl: transferred 1074.76kb in 20.817 seconds (51.6289 kb/s) |
Last edited by Kosh on Tue Mar 10, 2009 1:03 am; edited 1 time in total |
|
| Back to top |
|
|
Rollie
Site Admin
Joined: 28 Nov 2004
Posts: 5374
Location: Austin, TX
WR Updates: 480,131
Rollie WR Profile
|
| Posted: Wed Mar 04, 2009 11:29 am Post subject: |
|
|
| It should always give an Error if there is a problem |
|
| Back to top |
|
|
Kosh
Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile
|
| Posted: Thu Mar 05, 2009 11:40 pm Post subject: |
|
|
| Good deal, thanks |
|
| Back to top |
|
|
Ceto
Shady Dealer
Joined: 16 Oct 2005
Posts: 350
Location: Plymouth, NH
WR Updates: 193,167
Ceto WR Profile
|
|
| Back to top |
|
|
Kosh
Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile
|
| Posted: Tue Mar 10, 2009 12:56 am Post subject: |
|
|
Interesting. In my case, my WoW account name isn't the same as my account here, so it would require some tweaking for me. And of course, you can't assume where the game is installed on someone else's machine, but that's configurable (on my machine, it's at /Applications/Games/World of Warcraft/, for example).
Note that you could compress the file in Python, using the gzip module.
I'll shift into "UNIX security mode" briefly & point out that passing a password on the command-line is not secure, since it will show up in the Process Table (visible using "ps", etc.). Of course, on a machine you have complete physical control over, it's not a big deal. Just pointing it out for future reference.
/UNIX geek mode
Oh, I might as well point out that I discovered a bug in my "version 2" script; it doesn't successfully capture the exit status of both the "curl" and "awk" commands. Instead of reposting the whold script, I will just edit the script above, since there are only a couple of lines that changed (I figure everyone's eyes will glaze over if I post a "diff" ). |
|
| Back to top |
|
|
Kosh
Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile
|
| Posted: Thu May 07, 2009 10:18 pm Post subject: |
|
|
Well, I've made a couple more tweaks to my upload script. Most notably, it now passes any errors out of the script, so that they can be tested by other scripts (or a "while" loop, when necessary :p ). Other than that, I mainly consolidated the error reporting, including outputting the time on errors.
enjoy
| Code: | #!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
lastERR=0
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME: Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ $cpFN -nt $cpFNgz ] ); then
{
echo " lua file newer -- recompressing..."
gzip -c -9 $cpFN > $cpFNgz
lastERR=$?
if ( [ $lastERR -ne 0 ] ); then
{
errTEXT="gzip"
needUPLOAD=-1
}
else
needUPLOAD=1
fi
}
else
{
# need to force another upload?
if ( [ "$1" == "-f" -o -e $myREUPFLAG ] ); then
{
echo " compressed file up to date, but upload needed"
needUPLOAD=1
}
else
{
echo " compressed file up to date -- nothing to do."
needUPLOAD=0
}
fi
}
fi
if ( [ $needUPLOAD -gt 0 ] ); then
{
echo " Starting upload..."
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
-w "\n!!!curl %{size_upload} %{time_total} %{speed_upload}" \
http://www.warcraftrealms.com/uniupload.php \
| awk \
'/^$/ { next }
/^Checking your/
/^Found / { if( foundLine != "" ) foundLine = foundLine "; ";
if( $NF == "<foundLine> 0 )
{ kb = $2 / 1024;
print "\ncurl: transferred " kb "kb in " $3 " seconds (" kb / $3 " kb/s)";
}
}
/^Error/ { print "***\n" $0;
if( index( $0, "nable to connect" ) > 0 ) # no DB connection
exit 221;
if( index( $NF, "uniupload.php" ) > 0 ) # mystery SQL error
exit 222;
exit 220
}
'
# copy the array of pipe statuses - $lastERR refs [0] (curl stat)
lastERR=( "${PIPESTATUS[@]}" )
awkERR=${lastERR[1]} # (status of awk)
echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
if ( [ $lastERR -ne 0 ] ); then
{
errTEXT="curl"
# flag the failed upload, so we can retry later
touch $myREUPFLAG
}
else
{
if ( [ $awkERR -ne 0 ] ); then
{
errTEXT="server during upload"
lastERR=$awkERR
# flag the failed upload, so we can retry later
touch $myREUPFLAG
}
else
{
date "+ upload completed at %H:%M:%S"
rm -f $myREUPFLAG
}
fi
}
fi
}
fi
cd $holdPWD
if ( [ $lastERR -ne 0 ] ); then
{
echo -n "$myNAME: Error from $errTEXT: $lastERR"
date "+ [%H:%M:%S]***"
}
fi
exit $lastERR
#eof |
|
|
| Back to top |
|
|
bartman009
Joined: 13 Jun 2009
Posts: 1
WR Updates: 0
bartman009 WR Profile
|
| Posted: Sat Jun 13, 2009 11:21 pm Post subject: |
|
|
thanks a lot for updating the script!
assurance vie |
|
| Back to top |
|
|
Kosh
Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile
|
| Posted: Thu Jun 18, 2009 1:33 am Post subject: |
|
|
| You're welcome |
|
| Back to top |
|
|
Kosh
Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile
|
| Posted: Sat Jun 27, 2009 7:34 pm Post subject: |
|
|
After I discovered that the forum software is mangling any text between a "less-than" and a "greater-than" symbol (i.e., angle brackets), I thought I had better check this posting of my script. Sure enough, it is also mangled. Here's another copy with "~{~" and "~}~" substituted for all left and right angle brackets, respectively. To use, just reverse the substitution after copying.
Sorry about the confusion. I wouldn't expect phpBB to do any kind of (apparent) attempted HTML parsing inside a code block.
| Code: | #!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
lastERR=0
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME: Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ $cpFN -nt $cpFNgz ] ); then
{
echo " lua file newer -- recompressing..."
gzip -c -9 $cpFN ~}~ $cpFNgz
lastERR=$?
if ( [ $lastERR -ne 0 ] ); then
{
errTEXT="gzip"
needUPLOAD=-1
}
else
needUPLOAD=1
fi
}
else
{
# need to force another upload?
if ( [ "$1" == "-f" -o -e $myREUPFLAG ] ); then
{
echo " compressed file up to date, but upload needed"
needUPLOAD=1
}
else
{
echo " compressed file up to date -- nothing to do."
needUPLOAD=0
}
fi
}
fi
if ( [ $needUPLOAD -gt 0 ] ); then
{
echo " Starting upload..."
echo "~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~"
curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
-w "\n!!!curl %{size_upload} %{time_total} %{speed_upload}" \
http://www.warcraftrealms.com/uniupload.php \
| awk \
'/^$/ { next }
/^Checking your/
/^Found / { if( foundLine != "" ) foundLine = foundLine "; ";
if( $NF == "~{~=" )
foundLine = foundLine $(NF-2) " " $(NF-1);
else
foundLine = foundLine substr( $0, 7 );
next
}
/^Attempting / { if( foundLine != "" )
{ print "Found: " foundLine;
foundLine = "";
}
print $0;
next
}
/^Username/
/ will be credited /
/^!!!curl/ { if( $3 ~}~ 0 )
{ kb = $2 / 1024;
print "\ncurl: transferred " kb "kb in " $3 " seconds (" kb / $3 " kb/s)";
}
}
/^Error/ { print "***\n" $0;
if( index( $0, "nable to connect" ) ~}~ 0 ) # no DB connection
exit 221;
if( index( $NF, "uniupload.php" ) ~}~ 0 ) # mystery SQL error
exit 222;
exit 220
}
'
# copy the array of pipe statuses - $lastERR refs [0] (curl stat)
lastERR=( "${PIPESTATUS[@]}" )
awkERR=${lastERR[1]} # (status of awk)
echo "~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~"
if ( [ $lastERR -ne 0 ] ); then
{
errTEXT="curl"
# flag the failed upload, so we can retry later
touch $myREUPFLAG
}
else
{
if ( [ $awkERR -ne 0 ] ); then
{
errTEXT="server during upload"
lastERR=$awkERR
# flag the failed upload, so we can retry later
touch $myREUPFLAG
}
else
{
date "+ upload completed at %H:%M:%S"
rm -f $myREUPFLAG
}
fi
}
fi
}
fi
cd $holdPWD
if ( [ $lastERR -ne 0 ] ); then
{
echo -n "$myNAME: Error from $errTEXT: $lastERR"
date "+ [%H:%M:%S]***"
}
fi
exit $lastERR
#eof |
|
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
