.. _running-code: Running code on ZeroCloud ========================= These examples below include executing code using just plain old ``curl`` commands on the command line, as well as scripting using Python and the `requests `_ module. Jump to a section: - :ref:`Setup: Getting an auth token ` - :ref:`POST a Python script ` - :ref:`POST a ZeroVM image ` - :ref:`POST a job description to a ZeroVM application ` - :ref:`Run a ZeroVM application with an object GET ` - :ref:`MapReduce application ` .. _runningcode-setup: Setup: Getting an auth token ---------------------------- The first thing you need to do is get an auth token and find the storage URL for your account in Swift. For convenience, you can get this information simply by running ``zpm auth``: .. code-block:: bash $ zpm auth Auth token: PKIZ_Zrz_Qa5NJm44FWeF7Wp... Storage URL: http://127.0.0.1:8080/v1/AUTH_7fbcd8784f8843a180cf187bbb12e49c Setting a couple of environment variables with these values will make commands more concise and convenient to execute: .. code-block:: bash $ export OS_AUTH_TOKEN=PKIZ_Zrz_Qa5NJm44FWeF7Wp... $ export OS_STORAGE_URL=http://127.0.0.1:8080/v1/AUTH_7fbcd8784f8843a180cf187bbb12e49c .. _runningcode-postpyscript: POST a Python script -------------------- This is the simplest and easiest way to execute code on ZeroCloud. First, write the following the code into a file called ``example``. .. code-block:: python #!file://python2.7:python import sys print("Hello from ZeroVM!") print("sys.platform is '%s'" % sys.platform) Execute it using ``curl``: .. code-block:: bash $ curl -i -X POST -H "X-Auth-Token: $OS_AUTH_TOKEN" \ -H "X-Zerovm-Execute: 1.0" -H "Content-Type: application/python" \ --data-binary @example $OS_STORAGE_URL Using a Python script: .. code-block:: python import os import requests storage_url = os.environ.get('OS_STORAGE_URL') headers = { 'X-Zerovm-Execute': 1.0, 'X-Auth-Token': os.environ.get('OS_AUTH_TOKEN'), 'Content-Type': 'application/python', } with open('example') as fp: response = requests.post(storage_url, data=fp.read(), headers=headers) print(response.content) You can write and execute any Python code in this way, using any of the modules in the standard library. .. _runningcode-postzerovmimage: POST a ZeroVM image ------------------- Another way to execute code on ZeroCloud is to create a specially constructed tarball (a "ZeroVM image") and ``POST`` it directly to ZeroCloud. A "ZeroVM image" is a tarball with at minimum a ``boot/system.map`` file. The ``boot/system.map``, or job description, contains runtime execution information which tells ZeroCloud what to execute. This is useful if your code consists of multiple source files (not just a single script). You can pack everything into a single file and execute it. This method is also useful if you want to just execute something once, meaning that once ZeroCloud executes the application, the app is thrown away. In this example, we'll do just that. Create the following files: ``mymath.py``: .. code-block:: python def add(a, b): return a + b ``main.py``: .. code-block:: python import mymath a = 5 b = 6 the_sum = mymath.add(a, b) print("%s + %s = %s" % (a, b, the_sum)) Create a ``boot`` directory, then ``boot/system.map`` file: .. code-block:: javascript [{ "name": "example", "exec": { "path": "file://python2.7:python", "args": "main.py" }, "devices": [ {"name": "python2.7"}, {"name": "stdout"} ] }] Create the ZeroVM image: .. code-block:: bash $ tar cf example.tar boot/system.map main.py mymath.py Execute the ZeroVM image directly on ZeroCloud using ``curl``: .. code-block:: bash $ curl -i -X POST -H "Content-Type: application/x-tar" \ -H "X-Auth-Token: $OS_AUTH_TOKEN" -H "X-Zerovm-Execute: 1.0" \ --data-binary @example.tar $OS_STORAGE_URL Using a Python script: .. code-block:: python import os import requests storage_url = os.environ.get('OS_STORAGE_URL') headers = { 'X-Zerovm-Execute': 1.0, 'X-Auth-Token': os.environ.get('OS_AUTH_TOKEN'), 'Content-Type': 'application/x-tar', } with open('example.tar') as fp: response = requests.post(storage_url, data=fp.read(), headers=headers) print(response.content) .. _runningcode-postjobdesc: POST a job description to a ZeroVM application ---------------------------------------------- This method is useful if you want to execute the same application multiple times, for example, to run an application to process multiple different files. In this example, we will upload a packaged application into Swift and then subsequently POST job descriptions to execute the application. This can be done multiple times, and with different arguments. We'll use this to build a small application. Create a directory ``sampleapp`` and in it, create the following files: ``main.py``: .. code-block:: python import csv with open('/dev/input') as fp: reader = csv.reader(fp) for id, name, email, balance in reader: print('%(name)s: %(balance)s' % dict(name=name, balance=balance)) Create an ``example.tar`` containing the Python script: .. code-block:: bash $ tar cf example.tar main.py Create a container for the application: .. code-block:: bash $ swift post example Upload the image into Swift: .. code-block:: bash $ swift upload example example.tar Now we need to create a couple of files for the application to read and process. ``data1.csv``: .. code-block:: text id,name,email,balance 1,Alice,alice@example.com,1000 2,Bob,bob@example.com,-500 ``data2.csv``: .. code-block:: text id,name,email,balance 3,David,david@example.com,15000 4,Erin,erin@example.com,25000 Upload the data files into Swift: .. code-block:: bash $ swift upload example data1.csv data2.csv ``job.json``: .. code-block:: javascript [{ "name": "example", "exec": { "path": "file://python2.7:python", "args": "main.py" }, "devices": [ {"name": "python2.7"}, {"name": "stdout"}, {"name": "input", "path": "swift://~/example/data1.csv"}, {"name": "image", "path": "swift://~/example/example.tar"} ] }] Execute it using ``curl``: .. code-block:: bash $ curl -i -X POST -H "Content-Type: application/json" \ -H "X-Auth-Token: $OS_AUTH_TOKEN" -H "X-Zerovm-Execute: 1.0" \ --data-binary @job.json $OS_STORAGE_URL Execute it using a Python script: .. code-block:: python import os import requests storage_url = os.environ.get('OS_STORAGE_URL') headers = { 'X-Zerovm-Execute': 1.0, 'X-Auth-Token': os.environ.get('OS_AUTH_TOKEN'), 'Content-Type': 'application/json', } with open('job.json') as fp: response = requests.post(storage_url, data=fp.read(), headers=headers) print(response.content) You can process a different input file by simply changing the ``job.json`` and re-running the application (using ``curl`` or the Python script above). For example, change this line .. code-block:: text {"name": "input", "path": "swift://~/example/data1.csv"}, to this: .. code-block:: text {"name": "input", "path": "swift://~/example/data2.csv"}, Your ``job.json`` file should now look like this: .. code-block:: javascript [{ "name": "example", "exec": { "path": "file://python2.7:python", "args": "main.py" }, "devices": [ {"name": "python2.7"}, {"name": "stdout"}, {"name": "input", "path": "swift://~/example/data2.csv"}, {"name": "image", "path": "swift://~/example/example.tar"} ] }] Try running that and see the difference in the output: .. code-block:: bash $ curl -i -X POST -H "Content-Type: application/json" \ -H "X-Auth-Token: $OS_AUTH_TOKEN" -H "X-Zerovm-Execute: 1.0" \ --data-binary @job.json $OS_STORAGE_URL .. _runningcode-objectget: Run a ZeroVM application with an object GET ------------------------------------------- It is possible to attach applications to particular types of objects and run that application when the object is retrieved (using a GET request) from Swift. In this example, we'll write an application which processes JSON file objects and returns a pretty-printed version of the contents. The idea here is that we take some raw JSON data and make it more human-readable. Create the following files in a new directory ``sampleapp2``: ``data.json``: .. code-block:: javascript {"type": "GeometryCollection", "geometries": [{ "type": "Point", "coordinates": [100.0, 0.0]}, {"type": "LineString", "coordinates": [[101.0, 0.0], [102.0, 1.0]]}]} ``prettyprint.py``: .. code-block:: python import json import pprint with open('/dev/input') as fp: data = json.load(fp) print(pprint.pformat(data)) ``config``: .. code-block:: javascript [{ "name": "prettyprint", "exec": { "path": "file://python2.7:python", "args": "prettyprint.py" }, "devices": [ {"name": "python2.7"}, {"name": "stdout"}, {"name": "input", "path": "{.object_path}"}, {"name": "image", "path": "swift://~/example/prettyprint.tar"} ] }] Upload the test data: .. code-block:: bash $ swift post example # creates the container, if it doesn't exist already $ swift upload example data.json Bundle and upload the application: .. code-block:: bash $ tar cf prettyprint.tar prettyprint.py $ swift upload example prettyprint.tar Upload the configuration to a ``.zvm`` container: .. code-block:: bash $ swift post .zvm # creates the container, if it doesn't exist already $ swift upload .zvm config --object-name=application/json/config Now submit a GET request to the file, and it will be processed by the ``prettyprint`` application. Setting the ``X-Zerovm-Execute`` header to ``open/1.0`` is required to make this work. (Without this header you'll just get the raw file, unprocessed.) Using ``curl``: .. code-block:: bash $ curl -i -X GET $OS_STORAGE_URL/example/data.json \ -H "X-Zerovm-Execute: open/1.0" -H "X-Auth-Token: $OS_AUTH_TOKEN" Using a Python script: .. code-block:: python import os import requests storage_url = os.environ.get('OS_STORAGE_URL') headers = { 'X-Zerovm-Execute': 'open/1.0', 'X-Auth-Token': os.environ.get('OS_AUTH_TOKEN'), } response = requests.get(storage_url + '/example/data.json', headers=headers) print(response.content) .. _runningcode-mapreduce: MapReduce application --------------------- This example is a parallel wordcount application, constructed to utilize the MapReduce features of ZeroCloud. Create the project directory ++++++++++++++++++++++++++++ Create a directory for the project files. For example: .. code-block:: bash $ mkdir ~/mapreduce Then change into that directory: .. code-block:: bash $ cd ~/mapreduce Create Swift containers +++++++++++++++++++++++ We need to create two containers in Swift: one to hold our application data, and one to hold the application itself. .. code-block:: bash $ swift post mapreduce-data $ swift post mapreduce-app Upload sample data ++++++++++++++++++ Create a couple of text files and upload them into the ``mapreduce-data`` container. You can use the samples below, or any text you like. ``mrdata1.txt``: .. literalinclude:: mrdata1.txt ``mrdata2.txt``: .. literalinclude:: mrdata2.txt ``mrdata3.txt``: .. literalinclude:: mrdata3.txt Upload the files: .. code-block:: bash $ swift upload mapreduce-data mrdata1.txt $ swift upload mapreduce-data mrdata2.txt $ swift upload mapreduce-data mrdata3.txt Add ``zapp.yaml`` +++++++++++++++++ Add a ZeroVM application configuration template: .. code-block:: bash $ zpm new This will create a ``zapp.yaml`` file in the current directory. Open ``zapp.yaml`` in your favorite text editor. First, give the application a name, by changing the Change the ``execution`` section .. code-block:: yaml execution: groups: - name: "" path: file://python2.7:python args: "" devices: - name: python2.7 - name: stdout to look like this: .. code-block:: yaml :emphasize-lines: 11 execution: groups: - name: "map" path: file://python2.7:python args: "mapper.py" devices: - name: python2.7 - name: stdout - name: input path: "swift://~/mapreduce-data/*.txt" connect: ["reduce"] - name: "reduce" path: file://python2.7:python args: "reducer.py" devices: - name: python2.7 - name: stdout .. note:: The ``connect`` directive enables communication from the first execution group to the second. The creates a data pipeline where the results from the ``map`` execution, run on each text file in the ``mapreduce-data`` container, can be piped to the ``reduce`` part and combined into a single result. We also need to update the ``bundling`` section .. code-block:: yaml bundling: [] to include two Python source code files (which we will create below): .. code-block:: yaml bundling: ["mapper.py", "reducer.py"] The code ++++++++ Now let's write the code that will do our MapReduce wordcount. ``mapper.py``: .. literalinclude:: mrmapper.py ``reducer.py``: .. literalinclude:: mrreducer.py Bundle, deploy, and execute +++++++++++++++++++++++++++ Bundle: .. code-block:: bash $ zpm bundle created mapreduce.zapp Deploy: .. code-block:: bash $ zpm deploy mapreduce-app mapreduce.zapp Execute: .. code-block:: bash $ zpm execute mapreduce.zapp --container mapreduce-app 104 mapreduce-data/mrdata1.txt 101 mapreduce-data/mrdata2.txt 69 mapreduce-data/mrdata3.txt 274 total