NetworkQuantizer Class Reference

#include <NetworkQuantizer.hpp>

Inheritance diagram for NetworkQuantizer:

Public Member Functions
	NetworkQuantizer (INetwork *inputNetwork, const QuantizerOptions &options)
void	OverrideInputRange (LayerBindingId layerId, float min, float max) override
	Overrides the default quantization values for the input layer with the given id. More...
void	Refine (const InputTensors &inputTensors) override
	Refine input network with a set of refinement data for specified LayerBindingId. More...
std::pair< float, float >	GetMinMaxRange (LayerGuid guid, unsigned int idx)
INetworkPtr	ExportNetwork () override
	Extract final quantized network. More...

Additional Inherited Members
Static Public Member Functions inherited from INetworkQuantizer
static INetworkQuantizer *	CreateRaw (INetwork *inputNetwork, const QuantizerOptions &options=QuantizerOptions())
	Create Quantizer object and return raw pointer. More...
static INetworkQuantizerPtr	Create (INetwork *inputNetwork, const QuantizerOptions &options=QuantizerOptions())
	Create Quantizer object wrapped in unique_ptr. More...
static void	Destroy (INetworkQuantizer *quantizer)
	Destroy Quantizer object. More...
Protected Member Functions inherited from INetworkQuantizer
virtual	~INetworkQuantizer ()

Detailed Description

Definition at line 20 of file NetworkQuantizer.hpp.

Constructor & Destructor Documentation

NetworkQuantizer()

NetworkQuantizer ( INetwork *  inputNetwork, const QuantizerOptions &  options  )

inline

Definition at line 23 of file NetworkQuantizer.hpp.

References NetworkQuantizer::OverrideInputRange(), and NetworkQuantizer::Refine().

    : m_InputNetwork(inputNetwork),
      m_NetworkId(0),
      m_Runtime(nullptr, &IRuntime::Destroy),
      m_RefineCount(0),
      m_Options(options) {}

Member Function Documentation

ExportNetwork()

INetworkPtr ExportNetwork ( )

overridevirtual

Extract final quantized network.

Implements INetworkQuantizer.

Definition at line 137 of file NetworkQuantizer.cpp.

References armnn::QAsymmS8, armnn::QAsymmU8, armnn::QSymmS16, armnn::QSymmS8, Graph::TopologicalSort(), and armnn::VisitLayers().

Referenced by NetworkQuantizer::GetMinMaxRange().

{
    const Graph& graph = PolymorphicDowncast<const Network*>(m_InputNetwork)->GetGraph().TopologicalSort();

    // Step 1) Walk the graph and populate default min/max values for
    // intermediate tensors, only if Runtime does not exist (created
    // if Refine has been called)
    if (!m_Runtime)
    {
        m_Ranges.SetDynamicMode(false);
        StaticRangeVisitor rangeVisitor(m_Ranges);
        VisitLayers(graph, rangeVisitor);
    }
    else
    {
        // Set min/max range of non-calibrated layers to parent layer's range
        m_DynamicQuantizationVisitor.value().VisitNonCalibratedLayers();
        // now tear down the runtime and the dynamic visitor.
        m_Runtime.reset(nullptr);
        m_DynamicQuantizationVisitor = EmptyOptional();
        m_RefineCount = 0;
    }

    // Step 2) Convert input InputNetwork to Quantized InputNetwork
    std::unique_ptr<IQuantizationScheme> quantizationScheme;
    switch (m_Options.m_ActivationFormat)
    {
        case DataType::QAsymmU8:
            quantizationScheme = std::make_unique<QAsymmU8QuantizationScheme>();
            break;
        case DataType::QAsymmS8:
            quantizationScheme = std::make_unique<QAsymmS8QuantizationScheme>();
            break;
        case DataType::QSymmS8:
            quantizationScheme = std::make_unique<QSymmS8QuantizationScheme>();
            break;
        case DataType::QSymmS16:
            quantizationScheme = std::make_unique<QSymm16QuantizationScheme>();
            break;
        default:
            throw InvalidArgumentException("Unsupported quantization target");
    }

    QuantizerVisitor quantizerVisitor(m_Ranges, quantizationScheme.get(), m_Options.m_PreserveType);
    VisitLayers(graph, quantizerVisitor);

    // clear the ranges
    m_Ranges.Reset();

    return quantizerVisitor.RetrieveFinalNetwork();
}

GetMinMaxRange()

std::pair<float, float> GetMinMaxRange ( LayerGuid  guid, unsigned int  idx  )

inline

Definition at line 34 of file NetworkQuantizer.hpp.

References NetworkQuantizer::ExportNetwork(), and RangeTracker::GetRange().

{ return m_Ranges.GetRange(guid, idx); }

OverrideInputRange()

void OverrideInputRange ( LayerBindingId  layerId, float  min, float  max  )

overridevirtual

Overrides the default quantization values for the input layer with the given id.

Implements INetworkQuantizer.

Definition at line 51 of file NetworkQuantizer.cpp.

References Graph::GetInputLayers(), and armnn::VisitLayers().

Referenced by NetworkQuantizer::NetworkQuantizer().

{
    const Graph& graph = PolymorphicDowncast<const Network*>(m_InputNetwork)->GetGraph();
    auto inputLayers = graph.GetInputLayers();

    // Walk the input layers of the graph and override the quantization parameters of the one with the given id
    OverrideInputRangeVisitor overrideInputRangeVisitor(m_Ranges, layerId, RangeTracker::MinMaxRange{min, max});
    VisitLayers(inputLayers, overrideInputRangeVisitor);
}

Refine()

void Refine ( const InputTensors &  inputTensors )

overridevirtual

Refine input network with a set of refinement data for specified LayerBindingId.

Implements INetworkQuantizer.

Definition at line 61 of file NetworkQuantizer.cpp.

References IRuntime::Create(), armnnUtils::FindMinMax(), armnn::Optimize(), Graph::TopologicalSort(), and armnn::VisitLayers().

Referenced by NetworkQuantizer::NetworkQuantizer().

{
    // The first time Refine is called the m_Runtime and the DynamicQuantizationVisitor
    // will not have been created. Need to get the environment set up, Runtime loaded,
    // DynamicQuantizationVisitor created and run over the network to initialise itself
    // and the RangeTracker the Debug callback registered and an initial inference
    // done to set up the first min/max values
    if (!m_Runtime)
    {
        m_RefineCount = 0;
        m_Ranges.SetDynamicMode(true);
        const Graph& cGraph = PolymorphicDowncast<const Network*>(m_InputNetwork)->GetGraph().TopologicalSort();

        // need to insert Debug layers in the DynamicQuantizationVisitor
        Graph& graph = const_cast<Graph&>(cGraph);

        // Initialize RangeTracker to the default values for each layer.
        // The default values are overwritten by the min/max that is
        // recorded during the first dataset min/max calibration. This
        // initialisation is only required for the first call of Refine().
        m_DynamicQuantizationVisitor = DynamicQuantizationVisitor(m_Ranges, graph);
        VisitLayers(cGraph, m_DynamicQuantizationVisitor.value());

        IRuntime::CreationOptions options;
        m_Runtime = IRuntime::Create(options);

        // Optimize network - debug already enabled for layers that require quantization
        OptimizerOptions optimizerOptions(false, false);
        std::vector<BackendId> backends = {"CpuRef"};
        IOptimizedNetworkPtr optimizedNet = Optimize(*m_InputNetwork,
                                                     backends,
                                                     m_Runtime->GetDeviceSpec(),
                                                     optimizerOptions);

        m_Runtime->LoadNetwork(m_NetworkId, std::move(optimizedNet));

        // Debug callback function to refine min/max in RangeTracker
        auto rangeTrackerCallback = [&](LayerGuid guid, unsigned int slotIndex, ITensorHandle *tensorHandle) {
            // Get min/max pair from tensor data
            std::pair<float, float> minMax = armnnUtils::FindMinMax(tensorHandle);

            // For first calibration dataset, set min/max range in RangeTracker to
            // min/max ranges gathered during inference
            if (m_RefineCount == 0)
            {
                m_Ranges.ResetMinMax(guid, slotIndex, minMax.first, minMax.second);
            }
            else
            {
                // For every other calibration dataset, only set min/max range if the
                // values gathered are less than / greater than originally recorded.
                m_Ranges.RefineMin(guid, slotIndex, minMax.first);
                m_Ranges.RefineMax(guid, slotIndex, minMax.second);
            }
        };

        m_Runtime->RegisterDebugCallback(m_NetworkId, rangeTrackerCallback);
    }

    // Create output tensor for EnqueueWorkload
    std::vector<armnn::BindingPointInfo> outputBindings;
    auto outputLayers = m_DynamicQuantizationVisitor.value().GetOutputLayers();
    std::vector<TContainer> outputVectors;
    for (auto outputLayerBindingId : outputLayers)
    {
        auto outputTensorInfo = m_Runtime->GetOutputTensorInfo(m_NetworkId, outputLayerBindingId);
        outputBindings.push_back(std::make_pair(outputLayerBindingId, outputTensorInfo));
        outputVectors.push_back(std::vector<float>(outputTensorInfo.GetNumElements(), 0));
    }
    OutputTensors outputTensors = armnnUtils::MakeOutputTensors<TContainer>(outputBindings, outputVectors);

    // Execute EnqueueWorkload with calibration image
    m_Runtime->EnqueueWorkload(m_NetworkId, inputTensors, outputTensors);
    ++m_RefineCount;
}

---

The documentation for this class was generated from the following files:

• src/armnn/NetworkQuantizer.hpp
• src/armnn/NetworkQuantizer.cpp