2011-03-28
Version 0.0.6 Released!
Yet another release is out! This time around we add support for "input" jobs. Previously, whenever we wanted to run a job, we had to copy the input over from our local machine or it had to already be in the underlying datastore. But if you just wanted to place a file in the datastore for later, it wasn't do-able. But now it is! Just run this:
result = neptune( :type => "input", :local => "get_mapreduce_output.rb", :remote => "/neptune-testbin/testscript.rb", :storage => "gstorage", :EC2_ACCESS_KEY => "your access key", :EC2_SECRET_KEY => "your secret key", :S3_URL => "commondatastorage.googleapis.com" ) puts result
From our example above, we indicate where our local copy of the file is (here it's another piece of Neptune code) and where we should store it in the datastore (as these use the S3 naming convention, they should begin with a slash '/'). For the short-term, the bucket should already exist (this matters for Google Storage but not the others). This method call then returns a boolean value corresponding to whether or not the operation succeeded. So upgrade to Neptune 0.0.6 and check back soon for more updates!
2011-03-19
Version 0.0.5 Released!
And we have a new release out! Neptune 0.0.5 adds support for alternative storage backends to be used when storing the results of Neptune jobs. Before, we always stored the output of Neptune jobs in the underlying database that AppScale uses (dubbed 'AppDB' in AppScale-speak). Now, you can store the results to Amazon S3, Eucalyptus Walrus, and Google Storage automatically!
Two different ways are available to make use of this support. If you like, you can manually specify your credentials when you run each Neptune job:
output = neptune(
:type => "mpi",
:output => "/neptune-testbin/mpi-output4.txt",
:code => "cpi",
:nodes_to_use => 1,
:storage => "gstorage",
:EC2_ACCESS_KEY => "your access key",
:EC2_SECRET_KEY => "your secret key",
:S3_URL => "commondatastorage.googleapis.com"
)
puts "job started? #{output[:result]}"
puts "message = #{output[:msg]}"Alternatively, you can put your credentials in your environment (ala the Eucalyptus style) and Neptune will automatically pick them up:
output = neptune(
:type => "mpi",
:output => "/neptune-testbin/mpi-output4.txt",
:code => "cpi",
:nodes_to_use => 1,
:storage => "s3"
)
puts "job started? #{output[:result]}"
puts "message = #{output[:msg]}"For the moment, we don't have automated bucket creation when using Google Storage, so if you're using it, make sure to manually create your bucket ahead of time. We'll get it resolved soon!
The latest AppScale branch has the necessary support for Neptune 0.0.5, and when we release AppScale 1.5, it will have this support as well. Let us know if other storage backends would be preferable in your apps!
