Virtual env live task

Virtual en live task allows you to execute live task code in virtual environment. With this you can use a Pip, Conda or an Conda R environment. Virtual env live task takes a mandatory parameter that define the virtual environment.

As they use virtual environment to be executed, theses live task can only take File ,Folder or ResourceSet as input/output. Theses live task are not run in the context of a normal Task.

When you write the code of a virtual env live task , there are 2 important variables available in your code context :

• source_paths : type List[string]
• list of string containing the path of files and folders defined as inputs. The list size and order matches the inputs defined in the task.
• target_folders : typeList[string]
• This variable must be set containing a list of path of output files and folders. The list size and order matches the outputs defined in the task.

Pip env live task

This task will allow you to run a python script inside a pip virtual environment. This environment requires a PipFile, here is the documentation about it : https://pipenv-fork.readthedocs.io/en/latest/basics.html . Juste write the text in the Pipenv configuration.

Conda env live task

This task will allow you to run a python script inside a conda virtual environment. This environment requires an environment yaml file, here is the documentation about it : https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-file-manually. Just write the content of the yml file the Conda env configuration.

Conda R live task

This task will allow you to run a R script inside a conda virtual envrionment. This is using the same yaml environment file as Conda.

This task must have the r-base dependency in its environment.

Here is an example of the environment.yml file for Conda R live task

name: .venv
channels:
  - conda-forge
dependencies:
  - r-base

Develop an env live task

The live task config integrate a light IDE to develop the live task. If you know what you are doing, this might be enough, but we recommend developing the live task in a real IDE before integrating it to the lab. The live task is just a script with some constraints on input/output, you can easily code it outside the lab and import the code in your lab task. This will save you some time and frustration.

You can develop it in the Codelab, here is how we recommend to do it.

Ready to develop your live task ? Let's go 💪

Install dependencies manually

The easiest way to test an env live task is to install required dependencies directly in the Codelab so it is available in the notebook.

Create a virtual environment

You can also create a virtual environment and run Jupyter notebook inside this environment. This only works for Conda and R Conda env.

1. Create you environment manually. You must include the ipykernel package into your environment.
2. Run the notebook using this environment.

Write and test the code

Once you notebook is ready you can develop the live task. Simply declar a source_path and target_folder variables anec then you can write you env live task code !

Here is a env live task that uses pyjwt to encode the content of a file to a jwt token and write the result in a text file.

source_paths = ['/lab/user/data/example.txt']
# Start of live task
import jwt
import os
# read source_path file
with open(source_paths[0], "r", encoding="utf-8") as fp:
    data = fp.read()
encoded_jwt = jwt.encode({"text": data}, "secret", algorithm="HS256")
# write result
result_path =  "result.txt"
with open(result_path, "w", encoding="utf-8") as fp:
    fp.write(encoded_jwt)
target_paths = [result_path]
# End of live task

Let's break it down.

First we declare the source_paths variable, this is to initialize the code in the same context as an env live task.

Then we import the jwt module which is from the pyjwt package installed in the virtual env. After that we read the source_path file, encode its content using the jwt module and write the result in a file named result.txt in the current directory. Then we set the output paths in the target_paths variable.

Here is the environment.yml file :

name: .venv3
channels:
- conda-forge
dependencies:
- python=3.8
- pyjwt==2.3.0
- ipykernel # use for notebook

Error

For virtual env live task, If an exception occurred, the line number the exception tells will be shifted by 2 (if no parameters). So if you have an error tells you that there is a problem line 12, it correspond to line 10 of your code in live task. If there are parameters the shift will be increased by one for each parameter.