From 480e31b477f877d254f95d2984c1a5b60e068450 Mon Sep 17 00:00:00 2001 From: "erik.andersson@arm.com" Date: Wed, 25 Aug 2021 16:42:19 +0200 Subject: MLBEDSW-4996: Community bug reporting on ML Platform Created "BUGS.md" which details to the Vela Community how to issue bug report using the Maniphest Bug Tracker. Also added a reference to it in "README.md". Signed-off-by: erik.andersson@arm.com Change-Id: I0120a890c8447907e32de6b10a24eceade09df7d --- BUGS.md | 173 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 9 +++- 2 files changed, 180 insertions(+), 2 deletions(-) create mode 100644 BUGS.md diff --git a/BUGS.md b/BUGS.md new file mode 100644 index 00000000..c1c499e3 --- /dev/null +++ b/BUGS.md @@ -0,0 +1,173 @@ +# Vela Community Bug Reporting + +If the Vela community encounters a bug, then we highly encourage the community +to report it. When issuing a report, use the +[Maniphest bug tracker](https://developer.mlplatform.org/maniphest/) +that is part of Phabricator on ML Platform. This site enables any Vela user to +discuss and share any Vela related issue with the community. + +The only prerequisite for this process is a GitHub account, which will be used +when creating an account on ML platform. + +If you also have a fix for the bug you have identified, then we encourage +you to follow the [contributions guide](CONTRIBUTIONS.md). + +The bug reporting process in its entirety is detailed below. + +## Maniphest bug tracker + +ML Platform uses a bug tracking system known as +[Maniphest](https://developer.mlplatform.org/maniphest/), +which is part of the Phabricator tool set. + +On the Maniphest platform, one can add active bugs using the Tasks and Bugs +tab which will then be addressable by the Vela community. + +The system is accessible directly via the link above. Alternatively, one can +navigate to [ML Platform](https://mlplatform.org) and select the "Developers" +drop down menu followed by "Development Tracker", which will bring you to +Phabricator. In order to initiate a bug report, select "Tasks and Bugs". + +Anyone can browse the Maniphest bug tracker, since it is a public forum. +However, in order to file your own ticket you need to be logged in +to an account on ML Platform. These can be created by clicking "Register" at + and then linking a GitHub ID. + +### Proprietary or Sensitive Information + +As mentioned previously, ML Platform and the Maniphest Bug tracking system +are public forums. As such, anyone can see information posted to them. +Therefore, any proprietary information you wish to share should be done using +Arm's ML Model Review described below. + +## Bug Reporting Process + +The entire process of handling a bug is summarized in the following steps. + +1. Discovering the bug + * A user notices a discrepancy in the functionality of Vela. + This can be as simple as Vela crashing upon runtime, + where an error message is printed out for the user to read. + However, in certain cases the bug is much harder to spot. + For example, Vela might run gracefully but optimize a network incorrectly. + Thus the error will manifest in a silent error, such as incorrect outputs. + In this case, as much information about the system and settings + should be noted. This will be used later on when handling the root cause. + A sufficient documentation of the problem is crucial when dealing with + these silent errors. + +2. Sharing the existence of the bug + * In order to resolve the issue, its existence needs to be shared + with the Vela community. Even though the user could fix it by themselves + in the first step, it is important that is is shared with the community. + This step will be detailed in the following sections. + +3. The community tackles the problem + * When the bug has been properly detailed, the Vela community will try to + find a solution. During this phase, further information may be requested + or shared through the corresponding bug ticket. If a solution is found, + the ticket will be updated with the relevant information. + +As mentioned, the second step is detailed below. + +### Opening the form + +To add a new bug report, navigate to the +[Bug tracker](https://developer.mlplatform.org/maniphest/). + +Click the button "+ Create Task" or navigate to +[the following page](https://developer.mlplatform.org/maniphest/task/edit/form/default/) + +### Title + +Write a title that describes the problem in a short and meaningful way. +An example of a good title is "AssertionError in weight_compressor.py +when optimizing DS-CNN". An example of a poor title is "Vela crashes on Linux". + +### Assigned To + +Leave this field blank, the report will be assigned when work is started on it. + +### Status + +If you were able to resolve the bug before submission and this ticket is simply +a notification to the Vela Community, select the "Resolved" option. Also, +follow the [contributions guide](CONTRIBUTIONS.md) in order to apply your fix. +If the bug has not yet been resolved, select the "Open" option instead. + +### Description + +This section should contain as much information as possible, +in order to allow someone else to reproduce the issue. + +Please provide the following details: + +- Vela command line and options. + +- Configuration files. If these contain information that you +do not wish to make public then please upload them using Arm's +[ML Model Review](https://www.arm.com/resources/contact-us/ml-model-review) +process described below. + +- Version of Vela used. + +- Error messages or other console output from Vela. + +- Link to a publicly accessible copy of the input network. +If this is not possible then you will need to upload the network using Arm's +[ML Model Review](https://www.arm.com/resources/contact-us/ml-model-review) +process described below. + +- License files related to your network, if applicable. + +Attached to the description you can find a toolbox, which contains a button to +upload files. This shall not be used to upload any network files, instead use +[ML Model Review](https://www.arm.com/resources/contact-us/ml-model-review). + +### Visible To + +ML Platform and the Manifest Bug Tracking System are public forums. +As such, anyone can see information posted to them. + +Thus, any proprietary information you wish to share should be done using Arm's +[ML Model Review](https://www.arm.com/resources/contact-us/ml-model-review). +described below. + +Hence, the content of the report should be freely viewable to the public +and thus the "Public" option should be selected in the "Visility" field. + +### Editable by + +In this field, restrict the editing privileges to yourself and members of the +Vela Optimizer group. This is done by selecting "Subscribers" in the +"Object Policies" entry of the "Editable by" drop down menu. + +Proceed by navigating down to the "Subscribers" field and typing in both your +username and "Vela Optimizer". To clarify, your username and "Vela Optimizer" +shall be separate entries in the "Subscribers" field. + +### Tags + +Finally, add "Vela Optimizer" in the "Tags" field. This ensures that the bug +is directed to the Vela team, enabling a rapid triage of the problem. + +### Submitting the report + +Submit the report by clicking the "Create New Task" button. + +A confirmation e-mail will be sent out to the linked e-mail address. +Any further changes to the report will be sent out in the same way. + +## Arm's [ML Model Review](https://www.arm.com/resources/contact-us/ml-model-review) process. + +As mentioned earlier, proprietary or sensitive information should not be shared +on this page. +Instead, any required sensitive content should be sent directly to Arm via +Arm's ML Model Review process. + +For instance, if you can't find any publicly hosted examples of your network, +e.g. a network hosted on Model Zoo, upload your own network to +[ML Model Review](https://www.arm.com/resources/contact-us/ml-model-review). + +It is a simple process that requires you to complete a short online form +after which you will be sent instructions on how to upload your files. \ No newline at end of file diff --git a/README.md b/README.md index f8e57aaa..40d99452 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ embedded system. The tool will also generate performance estimates (EXPERIMENTAL) for the compiled model. -The tool has limited functionality for compiling a +The tool has limited functionality for compiling a [TOSA](https://git.mlplatform.org/tosa/specification.git/) neural network (EXPERIMENTAL). @@ -171,7 +171,7 @@ vela --optimise Performance my_model.tflite ``` 6) Compile a network while optimising for the fastest inference time possible, -with an upper bound for the SRAM usage. The memory limit is set in bytes, i.e. +with an upper bound for the SRAM usage. The memory limit is set in bytes, i.e. run the following example if one requires a limit of 300KB. ```bash @@ -242,6 +242,11 @@ operators supported in this release. Please see [Vela Testing](TESTING.md). +## Bug Reporting + +Please see [Vela Community Bug Reporting](BUGS.md) for a description of how to +report bugs. + ## Resources Additional useful information: -- cgit v1.2.1