MLECO-4065: Revising scripts to use Python3.923.05

Revising documentation and scripts to use Python3.9 explicitly
to remove the need for installing it system-wide that might
break a distro's desktop and pacakge manager utilities.

Change-Id: I683b55dd0243d0a726dc94eba2431005d4897c8c
Signed-off-by: Kshitij Sisodia <kshitij.sisodia@arm.com>

4 files changed, 60 insertions, 64 deletions

diff --git a/docs/sections/building.md b/docs/sections/building.md
index c7d69c8..7af3fdb 100644
--- a/docs/sections/building.md
+++ b/docs/sections/building.md
@@ -103,7 +103,9 @@ Before proceeding, it is *essential* to ensure that the following prerequisites
 
   > **Note:** If you have an older version of Python installed (< 3.9) see the
   > [Troubleshooting](./troubleshooting.md#how-to-update-python3-package-to-newer-version)
-  > for instruction on how to update it.
+  > for instruction on how to install and use it.
+  > **Note:** This scenario might be true if you are using an Arm Virtual Hardware instance.
+  > See the troubleshooting link above on how to set up the environment in this case.
 
 - The build system creates a Python virtual environment during the build process. Please make sure that Python virtual
   environment module is installed by running:
@@ -112,12 +114,6 @@ Before proceeding, it is *essential* to ensure that the following prerequisites
     python3 -m venv
     ```
 
-  > **Note:** If you are using an Arm Virtual Hardware instance then Python virtual environment is not installed by default.
-  > You will need to install it yourself by running the following command:
-  >   ```commandline
-  >    sudo apt install python3.8-venv
-  >    ```
-
 - The build system uses external Python libraries during the building process. Please make sure that the latest pip and
   libsndfile versions are installed.
 
@@ -362,7 +358,11 @@ for the default 128 MACs configuration of the Arm® *Ethos™-U55* NPU and for t
 Arm® *Ethos™-U65* NPU.
 
 > **Note:** This script requires Python version 3.9 or higher. Please make sure all [build prerequisites](./building.md#build-prerequisites)
-> are satisfied.
+> are satisfied. If your environment points to system installed Python3 that is an older version than 3.9, choose the
+> required version explicitly after installing it:
+> ```sh
+> python3.9 ./set_up_default_resources.py
+> ```
 >
 > **Note:** This script also installs required version of CMake into the virtual environment, which can be used by activating it.
 
diff --git a/docs/sections/troubleshooting.md b/docs/sections/troubleshooting.md
index c8bb7b3..e6b921b 100644
--- a/docs/sections/troubleshooting.md
+++ b/docs/sections/troubleshooting.md
@@ -164,45 +164,36 @@ python3 --version
    sudo apt-get install python3.9 python3.9-venv libpython3.9 libpython3.9-dev
    ```
 
-3. Update the `python3` alternatives (set as 1 your previous version displayed at step 1):
-
-   ```commandline
-   sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1
-   sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 2
-   ```
-
-4. At the prompt, update the configuration by selecting Python3.9 as the chosen default alternative:
-
-   ```commandline
-   sudo update-alternatives --config python3
-   ```
-
-5. Python3.9 is now set as default, you can check it by running:
-
-   ```commandline
-   python3 --version
-   ```
-
-   ```log
-   Python 3.9.0
-   ```
-
-> **Note:** After updating from Python3.6 to Python3.9 it may happen that the `gnome-terminal` or the relative
-> shortcuts don't work anymore.
-> If when opening it from XTerm with `gnome-terminal` the following error appear:
+3. Explicitly specify this Python when executing the set-up Python scripts. For example:
+
+    ```commandline
+    python3.9 ./set_up_default_resources.py
+    ```
+> **Note:**: We do not recommend updating the Python version system-wide as it might break
+> various desktop utilities for your OS distribution. However, if you are using a container
+> or any other sandboxed environment, you can choose to update it using the following steps.
+>
+> * Update the `python3` alternatives (set as 1 your previous version displayed at step 1):
+>
+>   ```commandline
+>   sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1
+>   sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 2
+>   ```
+>
+> * At the prompt, update the configuration by selecting Python3.9 as the chosen default alternative:
+>
+>   ```commandline
+>   sudo update-alternatives --config python3
+>   ```
+> * Python3.9 is now set as default, you can check it by running:
 >
-> ```log
-> Traceback (most recent call last):
->   File "/usr/bin/gnome-terminal", line 9, in <module>
->     from gi.repository import GLib, Gio
->   File "/usr/lib/python3/dist-packages/gi/__init__.py", line 42, in <module>
->     from . import _gi
-> ImportError: cannot import name '_gi' from partially initialized module 'gi' (most likely due to a circular import)
-> (/usr/lib/python3/dist-packages/gi/> __init__.py)
-> ```
+>   ```commandline
+>   python3 --version
+>   ```
 >
-> Modify the `gnome-terminal` script located in `/usr/bin/` and changing the environment (first line of the script)
-> from `#!/usr/bin/python3` to `#!/usr/bin/python3.6`.
+>   ```log
+>   Python 3.9.0
+>   ```
 
 Next section of the documentation: [Appendix](appendix.md).
 
@@ -240,11 +231,11 @@ subprocess.CalledProcessError: Command 'python3 -m venv env' returned non-zero e
 You can fix this error by installing Python virtual environment and removing the corrupted resources_downloaded folder.
 
 ```commandline
-sudo apt install python3.9-venv
 rm -r resources_downloaded
 ```
 
-You can then try rebuilding again e.g.
+Follow the instructions to [update Python3 package to 3.9 version](./troubleshooting.md#how-to-update-python3-package-to-newer-version)
+before attempting a rebuild with:
 
 ```commandline
 python3 ./build_default.py
diff --git a/scripts/cmake/source_gen_utils.cmake b/scripts/cmake/source_gen_utils.cmake
index 95b3ba4..6287cb6 100644
--- a/scripts/cmake/source_gen_utils.cmake
+++ b/scripts/cmake/source_gen_utils.cmake
@@ -284,14 +284,26 @@ function(setup_source_generator)
         return()
     endif ()
 
+    # If environment is not found, find the required Python version
+    # and create it.
+    find_package(Python3 3.9
+            COMPONENTS Interpreter
+            REQUIRED)
+
+    if (NOT Python3_FOUND)
+        message(FATAL_ERROR "Required version of Python3 not found!")
+    else()
+        message(STATUS "Python3 (v${Python3_VERSION}) found: ${Python3_EXECUTABLE}")
+    endif()
+
     message(STATUS "Configuring python environment at ${PYTHON}")
 
     execute_process(
-        COMMAND ${PY_EXEC} -m venv ${DEFAULT_VENV_DIR}
+        COMMAND ${Python3_EXECUTABLE} -m venv ${DEFAULT_VENV_DIR}
         RESULT_VARIABLE return_code
     )
     if (NOT return_code STREQUAL "0")
-        message(FATAL_ERROR "Failed to setup python3 environment. Return code: ${return_code}")
+        message(FATAL_ERROR "Failed to setup Python3 environment. Return code: ${return_code}")
     endif ()
 
     execute_process(
diff --git a/set_up_default_resources.py b/set_up_default_resources.py
index 740cfb8..dd702c7 100755
--- a/set_up_default_resources.py
+++ b/set_up_default_resources.py
@@ -23,6 +23,7 @@ import shutil
 import subprocess
 import sys
 import urllib.request
+import venv
 from argparse import ArgumentParser
 from argparse import ArgumentTypeError
 from collections import namedtuple
@@ -382,18 +383,16 @@ def set_up_resources(
 
     metadata_dict = dict()
     vela_version = "3.8.0"
-    py3_major_version_minimum = 3  # Python >= 3.9 is required
-    py3_minor_version_minimum = 9
+    py3_version_minimum = (3, 9)
 
     # Is Python minimum requirement matched?
     py3_version = sys.version_info
-    if (
-        py3_version.major < py3_major_version_minimum
-        or py3_version.minor < py3_minor_version_minimum
-    ):
+    if py3_version < py3_version_minimum:
         raise Exception(
             "ERROR: Python3.9+ is required, please see the documentation on how to update it."
         )
+    else:
+        logging.info(f"Using Python version: {py3_version}")
 
     setup_script_hash_verified = False
     setup_script_hash = get_md5sum_for_file(Path(__file__).resolve())
@@ -433,15 +432,9 @@ def set_up_resources(
     env_activate = str(env_path / "bin" / "activate")
 
     if not env_path.is_dir():
-        os.chdir(download_dir)
-        # Create the virtual environment.
-        command = f"python3 -m venv {env_dirname}"
-        call_command(command)
-        commands = ["pip install --upgrade pip", "pip install --upgrade setuptools"]
-        for c in commands:
-            command = f"{env_python} -m {c}"
-            call_command(command)
-        os.chdir(current_file_dir)
+        # Create the virtual environment using current interpreter's venv
+        # (not necessarily the system's Python3)
+        venv.create(env_dir=env_path, with_pip=True, upgrade_deps=True)
 
     # 1.3 Install additional requirements first, if a valid file has been provided
     if additional_requirements_file and os.path.isfile(additional_requirements_file):