doc update to use APIs in lpot.experimental rather than lpot

Author: ftian1

README.md

@@ -482,6 +482,8 @@ The MSE tuning strategy does not work with the PyTorch adaptor layer. This strat
 
 [LPOT v1.2](https://github.com/intel/lpot/tree/v1.2) introduces incompatible changes in user facing APIs. Please refer to [incompatible changes](./docs/incompatible_changes.md) to know which incompatible changes are made in v1.2.
 
+[LPOT v1.2.1](https://github.com/intel/lpot/tree/v1.2.1) solves this backward compatible issues introduced in v1.2 by moving new user facing APIs to lpot.experimental package and keep old one as is. Please refer to [introduction](./docs/introduction.md) to know the details of user-facing APIs.
+
 # Support
 
 Submit your questions, feature requests, and bug reports to the

docs/benchmark.md

@@ -21,7 +21,7 @@ evaluation:                                          # optional. required if use
           size: 256
         CenterCrop:
           size: 224
-        ToTensor:
+        ToTensor: {}
         Normalize:
           mean: [0.485, 0.456, 0.406]
           std: [0.229, 0.224, 0.225]
@@ -39,7 +39,7 @@ evaluation:                                          # optional. required if use
           size: 256
         CenterCrop:
           size: 224
-        ToTensor:
+        ToTensor: {}
         Normalize:
           mean: [0.485, 0.456, 0.406]
 ```
@@ -52,7 +52,7 @@ In this case, you should config your dataloader and lpot will construct an evalu
 
 ```python
 dataset = Dataset() #  dataset class that implement __getitem__ method or __iter__ method
-from lpot import Benchmark, common
+from lpot.experimental import Benchmark, common
 evaluator = Benchmark(config.yaml)
 evaluator.dataloader = common.DataLoader(dataset, batch_size=batch_size)
 # user can also register postprocess and metric, this is optional

docs/introduction.md

@@ -8,10 +8,25 @@ Intel® Low Precision Optimization Tool is an open-source Python library designe
 
 The API is intended to unify low-precision quantization interfaces cross multiple DL frameworks for the best out-of-the-box experiences.
 
-The API consists of below componenets:
+> **NOTE**
+>
+> LPOT is keeping improving the user-facing APIs for better user experience. 
+>
+> Now there are two sets of user-facing APIs. One is the default one supported from LPOT v1.0 for backward compatibility. Another one is the new APIs in lpot.experimental package.
+> We recommend user to use the one in lpot.experimental. All of examples have been updated to use this experimental APIs.
+>
+> The major differences between the default use-facing APIs and the experiemntal APIs are:
+>   1. The experimental APIs abstract `lpot.experimental.common.Model` concept to cover those cases whose weight and graph files are stored seperately.
+>   2. The experimental APIs unifiy the calling style of `Quantization`, `Pruning`, and `Benchmark` class by setting model, calibration dataloader, evaluation dataloader, metric through class attributes rather than passing as function inputs.
+>   3. The experimental APIs refine LPOT built-in transforms/datasets/metrics by unifying the APIs cross different framework backends.
+
+## Experimental user-facing APIs
+
+The experimental user-facing APIs consist of below components:
 
 ### quantization-related APIs
 ```python
+# lpot.experimental.Quantization
 class Quantization(object):
     def __init__(self, conf_fname):
         ...
@@ -48,7 +63,7 @@ class Quantization(object):
         ...
 
 ```
-The `conf_fname` parameter used in the class initialization is the path to user yaml configuration file. This is a yaml file that is used to control the entire tuning behavior.
+The `conf_fname` parameter used in the class initialization is the path to user yaml configuration file. This is a yaml file that is used to control the entire tuning behavior on the model.
 
 > **LPOT User YAML Syntax**
 >
@@ -58,14 +73,15 @@ The `conf_fname` parameter used in the class initialization is the path to user
 
 ```python
 # Typical Launcher code
-from lpot import Quantization, common
+from lpot.experimental import Quantization, common
 
 # optional if LPOT built-in dataset could be used as model input in yaml
 class dataset(object):
   def __init__(self, *args):
       ...
 
   def __getitem__(self, idx):
+      # return single sample and label tuple without collate. label should be 0 for label-free case
       ...
 
   def len(self):
@@ -77,9 +93,13 @@ class custom_metric(object):
         ...
 
     def update(self, predict, label):
+        # metric update per mini-batch
         ...
 
     def result(self):
+        # final metric calculation invoked only once after all mini-batch are evaluated
+        # return a scalar to lpot for accuracy-driven tuning.
+        # by default the scalar is higher-is-better. if not, set tuning.accuracy_criterion.higher_is_better to false in yaml.
         ...
 
 quantizer = Quantization(conf.yaml)
@@ -96,7 +116,7 @@ q_model = quantizer()
 q_model.save('/path/to/output/dir') 
 ```
 
-`model` attribute in `Quantization` class is an abstraction of model formats cross different frameworks. LPOT supports passing the path of `keras model`, `frozen pb`, `checkpoint`, `saved model`, `torch.nn.model`, `mxnet.symbol.Symbol`, `gluon.HybirdBlock`, and `onnx model` to instantiate a `lpot.common.Model()` class and set to `quantizer.model`.
+`model` attribute in `Quantization` class is an abstraction of model formats cross different frameworks. LPOT supports passing the path of `keras model`, `frozen pb`, `checkpoint`, `saved model`, `torch.nn.model`, `mxnet.symbol.Symbol`, `gluon.HybirdBlock`, and `onnx model` to instantiate a `lpot.experimental.common.Model()` class and set to `quantizer.model`.
 
 `calib_dataloader` and `eval_dataloader` attribute in `Quantization` class is used to setup a calibration dataloader by code. It is optional to set if user sets corresponding fields in yaml.
 
@@ -130,33 +150,14 @@ class Pruning(object):
     def __call__(self):
         ...
 
-    @property
-    def calib_dataloader(self):
-        ...
-
-    @property
-    def eval_dataloader(self):
-        ...
-
     @property
     def model(self):
         ...
 
-    @property
-    def metric(self):
-        ...
-
-    @property
-    def postprocess(self, user_postprocess):
-        ...
-
     @property
     def q_func(self):
         ...
 
-    @property
-    def eval_func(self):
-        ...
 ```
 
 This API is used to do sparsity pruning. Currently it is Proof-of-Concept, LPOT only supports `magnitude pruning` on PyTorch.
@@ -171,9 +172,32 @@ class Benchmark(object):
 
     def __call__(self):
         ...
+
+    @property
+    def model(self):
+        ...
+
+    @property
+    def metric(self):
+        ...
+
+    @property
+    def b_dataloader(self):
+        ...
+
+    @property
+    def postprocess(self, user_postprocess):
+        ...
 ```
 
 This API is used to measure the model performance and accuarcy. 
 
 For how to use this API, please refer to [Benchmark Document](./benchmark.md)
 
+## Default user-facing APIs
+
+The default user-facing APIs would exist for backward compatiblity from v1.0 release. User could refer to [v1.1 API](https://github.com/intel/lpot/blob/v1.1/docs/introduction.md) to understand how default user-facing APIs work.
+
+A [HelloWorld example](../examples/helloworld/tf_example6) using default user-facing APIs is provided for user reference.
+
+Full examples using default user-facing APIs could be found at [here](https://github.com/intel/lpot/tree/v1.1/examples).

docs/tuning_strategies.md

@@ -49,7 +49,7 @@ quantization:                                        # optional. tuning constrai
     scale_propagation_concat: True                   # optional. default value is True.
     first_conv_or_matmul_quantization: True          # optional. default value is True.
   calibration:
-    sampling_size: 1000, 2000                        # optional. default value is the size of whole dataset. used to set how many portions of calibration dataset is used. exclusive with iterations field.
+    sampling_size: 1000, 2000                        # optional. default value is 100. used to set how many samples should be used in calibration.
     dataloader:                                      # optional. if not specified, user need construct a q_dataloader in code for lpot.Quantization.
       dataset:
         TFRecordDataset:

docs/tutorial.md

@@ -27,18 +27,18 @@ LPOT has added built-in supports on popular dataloader/dataset and metric to eas
 
 LPOT also supports register custom dataset and custom metric by code. 
 
-As for model, LPOT abstract a common API, named as [lpot.common.Model](../lpot/common/model.py), to cover the case in which model, weight, and other necessary info, are separately stored. Please refer to [model](./model.md) to know how to use it.
+As for model, LPOT abstract a common API, named as [lpot.experimental.common.Model](../lpot/experimental/common/model.py), to cover the case in which model, weight, and other necessary info, are separately stored. Please refer to [model](./model.md) to know how to use it.
 
 Postprocess is treat as a specical transform by LPOT which is only needed when model output is mismatching with the expected input of LPOT built-in metrics. if user is using custom metric, the postprocess is not needed indeed as custom metric implementation need ensure it can handle model output correctly. On the other hand, the postprocess logic becomes part of custom metric implementation.
 
 Below is an example of how to enable LPOT on TensorFlow mobilenet_v1 with built-in dataloader, dataset and metric.
 
 ```python
 # main.py
-import lpot
-quantizer = lpot.Quantization('./conf.yaml')
-model = quantizer.model("./mobilenet_v1_1.0_224_frozen.pb")
-quantized_model = quantizer(model)
+from lpot.experimental import Quantization, common
+quantizer = Quantization('./conf.yaml')
+quantizer.model = common.Model("./mobilenet_v1_1.0_224_frozen.pb")
+quantized_model = quantizer()
 ```
 
 ```yaml
@@ -82,8 +82,7 @@ If user wants to use a dataset or metric which does not support by LPOT built-in
 
 ```python
 # main.py
-import lpot
-from lpot.metric import BaseMetric
+from lpot.experimental import Quantization, common
 
 class Dataset(object):
   def __init__(self):
@@ -97,7 +96,7 @@ class Dataset(object):
       return len(self.test_images)
 
 # Define a customized Metric function 
-class MyMetric(BaseMetric):
+class MyMetric(object):
   def __init__(self, *args):
       self.pred_list = []
       self.label_list = []
@@ -116,19 +115,19 @@ class MyMetric(BaseMetric):
       return correct_num / self.samples
 
 # Quantize with customized dataloader and metric
-quantizer = lpot.Quantization('./conf.yaml')
+quantizer = Quantization('./conf.yaml')
 dataset = Dataset()
-quantizer.metric = lpot.common.Metric(MyMetric)
-quantizer.calib_dataloader = lpot.common.DataLoader(dataset, batch_size=1)
-quantizer.eval_dataloader = lpot.common.DataLoader(dataset, batch_size=1)
-quantizer.model = lpot.common.Model('../models/simple_model')
+quantizer.metric = common.Metric(MyMetric)
+quantizer.calib_dataloader = common.DataLoader(dataset, batch_size=1)
+quantizer.eval_dataloader = common.DataLoader(dataset, batch_size=1)
+quantizer.model = common.Model('../models/simple_model')
 q_model = quantizer()
 ```
 Note: 
 
-In the customized dataset, the `__getitem__()` interface must be implemented. It returns the (image, label) pair.
+In the customized dataset, the `__getitem__()` interface must be implemented and return single sample and label. In this example, it returns the (image, label) pair. User could return (image, 0) for label-free case.
 
-In the customized metric, the update() function records the predict result of each mini-batch, the result() function would be invoked by LPOT at the end of evaluation to return a higher-is-better scalar to reflect model accuracy.
+In the customized metric, the update() function records the predict result of each mini-batch, the result() function would be invoked by LPOT at the end of evaluation to return a scalar to reflect model accuracy. By default, this scalar is higher-is-better. If this scalar returned from customerized metric is a lower-is-better value, `tuning.accuracy_criterion.higher_is_better` in yaml should be set to `False`.
 
 ```yaml
 # conf.yaml

examples/helloworld/tf_example1/README.md

@@ -10,12 +10,12 @@ The configuration will create a dataloader of Imagenet and it will do Bilinear r
 ```yaml
 quantization:                                        # optional. tuning constraints on model-wise for advance user to reduce tuning space.
   calibration:
-    sampling_size: 20                            # optional. default value is the size of whole dataset. used to set how many portions of calibration dataset is used. exclusive with iterations field.
+    sampling_size: 20                                # optional. default value is 100. used to set how many samples should be used in calibration.
     dataloader:
       batch_size: 1
       dataset:
         ImageRecord:
-          root: <DATASET>/TF_imagenet/val/         # NOTE: modify to calibration dataset location if needed
+          root: <DATASET>/TF_imagenet/val/           # NOTE: modify to calibration dataset location if needed
       transform:
         ParseDecodeImagenet:
         BilinearImagenet: 
@@ -35,7 +35,7 @@ evaluation:                                          # optional. required if use
       batch_size: 32
       dataset:
         ImageRecord:
-          root: <DATASET>/TF_imagenet/val/          # NOTE: modify to evaluation dataset location if needed
+          root: <DATASET>/TF_imagenet/val/           # NOTE: modify to evaluation dataset location if needed
       transform:
         ParseDecodeImagenet:
         BilinearImagenet: 

examples/helloworld/tf_example1/conf.yaml

@@ -19,7 +19,7 @@ model:                                               # mandatory. lpot uses this
 
 quantization:                                        # optional. tuning constraints on model-wise for advance user to reduce tuning space.
   calibration:
-    sampling_size: 20                                # optional. default value is the size of whole dataset. used to set how many portions of calibration dataset is used. exclusive with iterations field.
+    sampling_size: 20                                # optional. default value is 100. used to set how many samples should be used in calibration.
     dataloader:
       batch_size: 1
       dataset:

examples/helloworld/tf_example2/README.md

@@ -55,8 +55,7 @@ class Dataset(object):
 ### 3.Define a customized metric  
 This customized metric will caculate accuracy.
 ```python
-from lpot.metric import Metric
-class MyMetric(Metric):
+class MyMetric(object):
   def __init__(self, *args):
       self.pred_list = []
       self.label_list = []

examples/helloworld/tf_example3/README.md

@@ -18,12 +18,12 @@ The configuration will help user to create a dataloader of Imagenet and it will
 ```yaml
 quantization:                                        # optional. tuning constraints on model-wise for advance user to reduce tuning space.
   calibration:
-    sampling_size: 20, 50                            # optional. default value is the size of whole dataset. used to set how many portions of calibration dataset is used. exclusive with iterations field.
+    sampling_size: 20, 50                            # optional. default value is 100. used to set how many samples should be used in calibration.
     dataloader:
       batch_size: 10
       dataset:
         ImageRecord:
-          root: /path/to/imagenet/         # NOTE: modify to calibration dataset location if needed
+          root: /path/to/imagenet/                   # NOTE: modify to calibration dataset location if needed
       transform:
         ParseDecodeImagenet:
         BilinearImagenet: 
@@ -39,7 +39,7 @@ evaluation:                                          # optional. required if use
       last_batch: discard 
       dataset:
         ImageRecord:
-          root: /path/to/imagenet/          # NOTE: modify to evaluation dataset location if needed
+          root: /path/to/imagenet/                   # NOTE: modify to evaluation dataset location if needed
       transform:
         ParseDecodeImagenet:
         BilinearImagenet: 
@@ -51,22 +51,9 @@ evaluation:                                          # optional. required if use
 3. Run quantizaiton
 * In order to do quanzation for slim models, we need to get graph from slim .ckpt first. 
 ```python
-    from lpot.experimental import Quantization,  common
+    from lpot.experimental import Quantization, common
     quantizer = Quantization('./conf.yaml')
 
-    # Get graph from slim checkpoint
-    from tf_slim.nets import inception
-    model_func = inception.inception_v1
-    arg_scope = inception.inception_v1_arg_scope()
-    kwargs = {'num_classes': 1001}
-    inputs_shape = [None, 224, 224, 3]
-    images = tf.compat.v1.placeholder(name='input', \
-    dtype=tf.float32, shape=inputs_shape)
-
-    from lpot.adaptor.tf_utils.util import get_slim_graph
-    graph = get_slim_graph('./inception_v1.ckpt', model_func, \
-            arg_scope, images, **kwargs)
-    
     # Do quantization
     quantizer.model = common.Model('./inception_v1.ckpt')
     quantized_model = quantizer()

examples/helloworld/tf_example3/conf.yaml

@@ -19,7 +19,7 @@ model:                                               # mandatory. lpot uses this
 
 quantization:                                        # optional. tuning constraints on model-wise for advance user to reduce tuning space.
   calibration:
-    sampling_size: 20                                # optional. default value is the size of whole dataset. used to set how many portions of calibration dataset is used. exclusive with iterations field.
+    sampling_size: 20                                # optional. default value is 100. used to set how many samples should be used in calibration.
     dataloader:
       batch_size: 10
       dataset: