WarcraftReamls.com
  FAQFAQ    SearchSearch    MemberlistMemberlist    UsergroupsUsergroups   RegisterRegister 
  ProfileProfile    Log in to check your private messagesLog in to check your private messages    Log inLog in 
Uploading (compressed) data vie shell script

 
Post new topic   This topic is locked: you cannot edit posts or make replies.    WarcraftRealms.com Forum Index -> WarcraftRealms Developers
View previous topic :: View next topic  
Author Message
Kosh
Census Taker


Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile

PostPosted: Thu Feb 12, 2009 11:29 pm    Post subject: Uploading (compressed) data vie shell script Reply with quote

I am guessing that the audience for a shell script to upload Census results is probably limited, particularly if your OS doesn't have a sufficiently UNIX-like shell environment, but I'll post what i have so far, anyway. Wink

Take the following code and save it as "censusup.sh" (or any name of your choosing) in a convenient directory that you have full permissions to. I put it in my normal "downloads" directory, because I already have a symlink to my CensusPlus.lua file there, but someplace like your home directory would also be fine. Note that you should not put it somewhere in the WoW directory tree. Oh yes, don't forget to make the script executable (chmod +x censusup.sh) Smile
Code:
#!/bin/sh
myNAME=`basename $0`
cpNAME=**your username**
cpPW='**your password**'
echo "$myNAME: Checking for new version of CencusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ CensusPlus.lua -nt CensusPlus.lua.gz ] );
  then {
    echo "  lua file newer -- recompressing...";
    gzip -c -9 CensusPlus.lua > CensusPlus.lua.gz;
    lastERR=$?
    if ( [ $lastERR -ne 0 ] );
      then {
        echo "$myNAME: Error from gzip: $lastERR"
        };
      else {
        echo "  ...compressed.  Starting upload..."
        echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
        curl -F CensusPlus=@CensusPlus.lua.gz -F user=$cpNAME -F pw=$cpPW -w "\ncurl: transfered %{size_upload} bytes in %{time_total} seconds (%{speed_upload} b/s)\n" http://www.warcraftrealms.com/uniupload.php
        lastERR=$?
        echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
        if ( [ $lastERR -ne 0 ] );
          then {
            echo "$myNAME: Error from curl: $lastERR"
            };
          else {
            echo "  upload successful."
            };
        fi
        };
    fi
    };
  else {
    echo "  compressed file up to date -- doing nothing.";
    };
fi
cd $holdPWD
#eof
(Note that the "curl" line is one long line.)

You will need to put your warcraftrealms.com username and password in the third and fourth lines. Keep the single quotes around the password, if you have any "special" characters in it.

Note that the script creates the gzip-compressed copy of the CensusPlus.lua file in the same directory as the script. It also assumes that the lua file (or a link to it) is also in the same directory. You can change this by hard-coding the path to the file in line 12, but I prefer having the symbolic link in the directory for convenience.

Note that I haven't spent a lot of time on this, so the error handling is rather simplistic (it just stops on errors), and it's fairly verbose. I also haven't gotten around to attempting to parse the response from the php script to look for logical errors. It just dumping out everything that's returned, so you need to look over the response to make sure that the "your name has been credited" line is there, and that no errors are reported.
Back to top
View user's profile Send private message
Rollie
Site Admin


Joined: 28 Nov 2004
Posts: 5374
Location: Austin, TX
WR Updates: 480,131
Rollie WR Profile

PostPosted: Fri Feb 13, 2009 9:22 am    Post subject: Reply with quote

well isn't that spiffy =)
_________________
phpbb:phpinfo()
Back to top
View user's profile Send private message Visit poster's website
Kosh
Census Taker


Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile

PostPosted: Mon Feb 16, 2009 10:44 pm    Post subject: Reply with quote

Yep, I think so Wink

A bonus to me is that I don't have to worry about hitting my head on the size limit (at least for quite a while), since I'm sending it compressed.
Back to top
View user's profile Send private message
Kosh
Census Taker


Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile

PostPosted: Wed Mar 04, 2009 1:12 am    Post subject: Reply with quote

Well, I put a little more work into my script. In addition to some cleanup, it now remembers if the compress succeeded but the upload failed, so a rerun of the script will retry the upload even if the lua file hasn't changed. It also strips down the server output (via awk) and also detects "logical" errors reported by the server-side script.

BTW Rollie, do any of the possible errors from uniupload.php start out with something other than "Error"? If so, then I will need to tweak the awk script.

Anyway, here's the updated version of censusup.sh:
Code:
#!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME: Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ $cpFN -nt $cpFNgz ] ); then
   {
   echo "  lua file newer -- recompressing...";
   gzip -c -9 $cpFN > $cpFNgz;
   lastERR=$?;
   if ( [ $lastERR -ne 0 ] ); then
      {
      echo "$myNAME: Error from gzip: $lastERR";
      needUPLOAD=-1;
      };
   else
      needUPLOAD=1;
   fi
   };
else
   {
# need to force another upload?
   if ( [ "$1" == "-f" -o -e $myREUPFLAG ] ); then
      {
      echo "  compressed file up to date, but upload needed";
      needUPLOAD=1;
      };
   else
      {
      echo "  compressed file up to date -- doing nothing.";
      needUPLOAD=0;
      };
   fi
   };
fi
if ( [ $needUPLOAD -gt 0 ] ); then
   {
   echo "  Starting upload..."
   echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
   curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
        -w "\n!!!curl %{size_upload} %{time_total} %{speed_upload}" \
        http://www.warcraftrealms.com/uniupload.php \
        | awk \
'/^$/           { next }
/^Checking your/
/^Found /      { if( foundLine != "" ) foundLine = foundLine "; ";
                 if( $NF == "<=" )
                    foundLine = foundLine $(NF-2) " " $(NF-1);
                 else
                    foundLine = foundLine substr( $0, 7 );
                 next
               }
/^Attempting / { if( foundLine != "" )
                    { print "Found: " foundLine;
                      foundLine = "";
                    }
                  print $0;
                  next
               }
/^Username/
/ will be credited /
/^!!!curl/     { kb = $2 / 1024;
                 print "\ncurl: transferred " kb "kb in " $3 " seconds (" kb / $3 " kb/s)";
               }
/^Error/       { print "***\n" $0; exit 10 }
'
# copy the array of pipe statuses - $lastERR refs [0] (curl stat)
   lastERR=( "${PIPESTATUS[@]}" )
   awkERR=${lastERR[1]}         # (status of awk)
   echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
   if ( [ $lastERR -ne 0 ] ); then
      {
      echo "$myNAME: Error from curl: $lastERR"
# flag the failed upload, so we can retry later
      touch $myREUPFLAG
      };
   else
      {
      if ( [ $awkERR -ne 0 ] ); then
         {
         echo "$myNAME: Error from server during upload"
# flag the failed upload, so we can retry later
         touch $myREUPFLAG
         };
      else
         {
         date "+  upload completed at %H:%M:%S"
         rm -f $myREUPFLAG
         };
      fi
      };
   fi
   };
fi
cd $holdPWD
#eof


The script woudl be a bit cleaner if I put the awk script in it's own file, but then another file would have to be floating around. Note that the 22 lines after the fourth line of the "curl" command is one long string constant (the awk command script).

In case you're wondering, here's an example of what the awk-massaged output looks like:
Code:
Checking your file
Found: CensusPlus Database; CensusPlus Info Data; Version: 4.2.2; Locale: US
Attempting to make entry

Username : Kosh<=
Kosh will be credited with this submission!

curl: transferred 1074.76kb in 20.817 seconds (51.6289 kb/s)


Last edited by Kosh on Tue Mar 10, 2009 1:03 am; edited 1 time in total
Back to top
View user's profile Send private message
Rollie
Site Admin


Joined: 28 Nov 2004
Posts: 5374
Location: Austin, TX
WR Updates: 480,131
Rollie WR Profile

PostPosted: Wed Mar 04, 2009 11:29 am    Post subject: Reply with quote

It should always give an Error if there is a problem
_________________
phpbb:phpinfo()
Back to top
View user's profile Send private message Visit poster's website
Kosh
Census Taker


Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile

PostPosted: Thu Mar 05, 2009 11:40 pm    Post subject: Reply with quote

Good deal, thanks Smile
Back to top
View user's profile Send private message
Ceto
Shady Dealer


Joined: 16 Oct 2005
Posts: 350
Location: Plymouth, NH
WR Updates: 193,167
Ceto WR Profile

PostPosted: Sun Mar 08, 2009 10:12 pm    Post subject: Reply with quote

I'll share mine, for comparison. It does not submit compressed data, and it's written in Python, with paths hard-coded for Mac OS X:

http://static.bwerp.net/~adam/2009/03/09/wr-upload.zip
_________________
Back to top
View user's profile Send private message Visit poster's website
Kosh
Census Taker


Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile

PostPosted: Tue Mar 10, 2009 12:56 am    Post subject: Reply with quote

Interesting. In my case, my WoW account name isn't the same as my account here, so it would require some tweaking for me. And of course, you can't assume where the game is installed on someone else's machine, but that's configurable (on my machine, it's at /Applications/Games/World of Warcraft/, for example).

Note that you could compress the file in Python, using the gzip module.

I'll shift into "UNIX security mode" briefly & point out that passing a password on the command-line is not secure, since it will show up in the Process Table (visible using "ps", etc.). Of course, on a machine you have complete physical control over, it's not a big deal. Just pointing it out for future reference. Smile

/UNIX geek mode


Oh, I might as well point out that I discovered a bug in my "version 2" script; it doesn't successfully capture the exit status of both the "curl" and "awk" commands. Instead of reposting the whold script, I will just edit the script above, since there are only a couple of lines that changed (I figure everyone's eyes will glaze over if I post a "diff" Wink ).
Back to top
View user's profile Send private message
Kosh
Census Taker


Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile

PostPosted: Thu May 07, 2009 10:18 pm    Post subject: Reply with quote

Well, I've made a couple more tweaks to my upload script. Most notably, it now passes any errors out of the script, so that they can be tested by other scripts (or a "while" loop, when necessary :p ). Other than that, I mainly consolidated the error reporting, including outputting the time on errors.

enjoy Smile
Code:
#!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
lastERR=0
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME: Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ $cpFN -nt $cpFNgz ] ); then
   {
   echo "  lua file newer -- recompressing..."
   gzip -c -9 $cpFN > $cpFNgz
   lastERR=$?
   if ( [ $lastERR -ne 0 ] ); then
      {
      errTEXT="gzip"
      needUPLOAD=-1
      }
   else
      needUPLOAD=1
   fi
   }
else
   {
# need to force another upload?
   if ( [ "$1" == "-f" -o -e $myREUPFLAG ] ); then
      {
      echo "  compressed file up to date, but upload needed"
      needUPLOAD=1
      }
   else
      {
      echo "  compressed file up to date -- nothing to do."
      needUPLOAD=0
      }
   fi
   }
fi
if ( [ $needUPLOAD -gt 0 ] ); then
   {
   echo "  Starting upload..."
   echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
   curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
        -w "\n!!!curl %{size_upload} %{time_total} %{speed_upload}" \
        http://www.warcraftrealms.com/uniupload.php \
        | awk \
'/^$/           { next }
/^Checking your/
/^Found /      { if( foundLine != "" ) foundLine = foundLine "; ";
                 if( $NF == "<foundLine> 0 )
                    { kb = $2 / 1024;
                      print "\ncurl: transferred " kb "kb in " $3 " seconds (" kb / $3 " kb/s)";
                    }
               }
/^Error/       { print "***\n" $0;
                 if( index( $0, "nable to connect" ) > 0 )  # no DB connection
                    exit 221;
                 if( index( $NF, "uniupload.php" ) > 0 )    # mystery SQL error
                    exit 222;
                 exit 220
               }
'
# copy the array of pipe statuses - $lastERR refs [0] (curl stat)
   lastERR=( "${PIPESTATUS[@]}" )
   awkERR=${lastERR[1]}         # (status of awk)
   echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
   if ( [ $lastERR -ne 0 ] ); then
      {
      errTEXT="curl"
# flag the failed upload, so we can retry later
      touch $myREUPFLAG
      }
   else
      {
      if ( [ $awkERR -ne 0 ] ); then
         {
         errTEXT="server during upload"
         lastERR=$awkERR
# flag the failed upload, so we can retry later
         touch $myREUPFLAG
         }
      else
         {
         date "+  upload completed at %H:%M:%S"
         rm -f $myREUPFLAG
         }
      fi
      }
   fi
   }
fi
cd $holdPWD
if ( [ $lastERR -ne 0 ] ); then
   {
   echo -n "$myNAME: Error from $errTEXT: $lastERR"
   date "+ [%H:%M:%S]***"
   }
fi
exit $lastERR
#eof
Back to top
View user's profile Send private message
bartman009



Joined: 13 Jun 2009
Posts: 1

WR Updates: 0
bartman009 WR Profile

PostPosted: Sat Jun 13, 2009 11:21 pm    Post subject: Reply with quote

thanks a lot for updating the script!
assurance vie
Back to top
View user's profile Send private message
Kosh
Census Taker


Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile

PostPosted: Thu Jun 18, 2009 1:33 am    Post subject: Reply with quote

You're welcome Smile
Back to top
View user's profile Send private message
Kosh
Census Taker


Joined: 01 Jul 2007
Posts: 84
Location: Somewhere on or near Earth
WR Updates: 1,210,548
Kosh WR Profile

PostPosted: Sat Jun 27, 2009 7:34 pm    Post subject: Reply with quote

After I discovered that the forum software is mangling any text between a "less-than" and a "greater-than" symbol (i.e., angle brackets), I thought I had better check this posting of my script. Sure enough, it is also mangled. Here's another copy with "~{~" and "~}~" substituted for all left and right angle brackets, respectively. To use, just reverse the substitution after copying.

Sorry about the confusion. I wouldn't expect phpBB to do any kind of (apparent) attempted HTML parsing inside a code block.

Code:
#!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
lastERR=0
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME: Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ $cpFN -nt $cpFNgz ] ); then
   {
   echo "  lua file newer -- recompressing..."
   gzip -c -9 $cpFN ~}~ $cpFNgz
   lastERR=$?
   if ( [ $lastERR -ne 0 ] ); then
      {
      errTEXT="gzip"
      needUPLOAD=-1
      }
   else
      needUPLOAD=1
   fi
   }
else
   {
# need to force another upload?
   if ( [ "$1" == "-f" -o -e $myREUPFLAG ] ); then
      {
      echo "  compressed file up to date, but upload needed"
      needUPLOAD=1
      }
   else
      {
      echo "  compressed file up to date -- nothing to do."
      needUPLOAD=0
      }
   fi
   }
fi
if ( [ $needUPLOAD -gt 0 ] ); then
   {
   echo "  Starting upload..."
   echo "~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~"
   curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
        -w "\n!!!curl %{size_upload} %{time_total} %{speed_upload}" \
        http://www.warcraftrealms.com/uniupload.php \
        | awk \
'/^$/           { next }
/^Checking your/
/^Found /      { if( foundLine != "" ) foundLine = foundLine "; ";
                 if( $NF == "~{~=" )
                    foundLine = foundLine $(NF-2) " " $(NF-1);
                 else
                    foundLine = foundLine substr( $0, 7 );
                 next
               }
/^Attempting / { if( foundLine != "" )
                    { print "Found: " foundLine;
                      foundLine = "";
                    }
                  print $0;
                  next
               }
/^Username/
/ will be credited /
/^!!!curl/     { if( $3 ~}~ 0 )
                    { kb = $2 / 1024;
                      print "\ncurl: transferred " kb "kb in " $3 " seconds (" kb / $3 " kb/s)";
                    }
               }
/^Error/       { print "***\n" $0;
                 if( index( $0, "nable to connect" ) ~}~ 0 )  # no DB connection
                    exit 221;
                 if( index( $NF, "uniupload.php" ) ~}~ 0 )    # mystery SQL error
                    exit 222;
                 exit 220
               }
'
# copy the array of pipe statuses - $lastERR refs [0] (curl stat)
   lastERR=( "${PIPESTATUS[@]}" )
   awkERR=${lastERR[1]}         # (status of awk)
   echo "~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~"
   if ( [ $lastERR -ne 0 ] ); then
      {
      errTEXT="curl"
# flag the failed upload, so we can retry later
      touch $myREUPFLAG
      }
   else
      {
      if ( [ $awkERR -ne 0 ] ); then
         {
         errTEXT="server during upload"
         lastERR=$awkERR
# flag the failed upload, so we can retry later
         touch $myREUPFLAG
         }
      else
         {
         date "+  upload completed at %H:%M:%S"
         rm -f $myREUPFLAG
         }
      fi
      }
   fi
   }
fi
cd $holdPWD
if ( [ $lastERR -ne 0 ] ); then
   {
   echo -n "$myNAME: Error from $errTEXT: $lastERR"
   date "+ [%H:%M:%S]***"
   }
fi
exit $lastERR
#eof
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   This topic is locked: you cannot edit posts or make replies.    WarcraftRealms.com Forum Index -> WarcraftRealms Developers All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
WarcraftRealms.com  


Powered by phpBB © 2001, 2005 phpBB Group