Noob question about the Change Detection notebook and using the model

[MarkoTukiainen]

Thanks for the updated address for the Change Detection notebook. I was able to port the code to be compatible with the current version, and was able to train some sort of a (dumb) model with it.

I'm afraid I'm very unfamiliar with python, pytorch and rastervision - but one's got to start somewhere?

Anyway, using the pipeline process to create predictions I'm getting some output from the validation cities (so this one):
rastervision run inprocess "config2.py" predict eval -a out_dir "blog_1/"

This creates labels.tif files which I can open in QGIS, and while the colors are reversed, the predictions are still there and usable.

However I'm kind of stumped as to how to a) either use the pipeline process for some specific images or b) use the model in the library mode to do the same. I've scoured the documentation, examples and the discussions but so far I've just come up with various slightly incompatible ways of doing something in that vein, and of course there's the fact that this is for change detection (as opposed to say object detection) - so I'll need to input two images (or a multirastersource?). And of course there's the whole 13 channels per image thing.

For the pipeline mode I run into errors immediately when trying to call rastervision pipeline with the model-bundle parameter and an infile and an outfile. (I even gdal_merged the 13 bands to a single geotiff to try this.) It always complains that "Could not read /opt/data/tmp/tmp_jr8nt_p/bundle/pipeline-config.json" or something similar.

For the library mode, here's what I've tried. I'm trying to detect changes in the beirut images. This of course does not work:

from pathlib import Path

root = Path('/opt/src/oscd')
img_dir = root / 'images'
train_labels_dir = root / 'trainlabels'
out_dir = '/opt/src/oscd/change-detection'
bundle_uri = '/opt/src/blog_3/train/model-bundle.zip'

from rastervision.pytorch_learner import SemanticSegmentationLearner

learner = SemanticSegmentationLearner.from_model_bundle(bundle_uri, training=False)

from rastervision.core.data import RasterioSource, MultiRasterSource

img_1_paths = sorted((img_dir / 'beirut' / 'imgs_1/').glob('*.tif'))
raster_sources_img_1 = [RasterioSource(uris=[str(path)]) for path in img_1_paths]

img_2_paths = sorted((img_dir / 'beirut' / 'imgs_2/').glob('*.tif'))
raster_sources_img_2 = [RasterioSource(uris=[str(path)]) for path in img_2_paths]

multi_rs_26 = MultiRasterSource(
    raster_sources=raster_sources_img_1 + raster_sources_img_2,
    primary_source_idx=1)

from rastervision.core.data import StatsTransformer
from rastervision.core.raster_stats import RasterStats

stats = RasterStats()
stats.compute(raster_sources=[multi_rs_26])
stats_transformer = StatsTransformer.from_raster_stats(stats=stats)

from rastervision.core.data import ClassConfig

class_config = ClassConfig(
    names=['no_change', 'change'], 
    colors=['lightgray', 'darkred'],
    null_class='no_change'
)

multi_rs_26_normalized = MultiRasterSource(
    raster_sources=raster_sources_img_1 + raster_sources_img_2,
    primary_source_idx=1,
    raster_transformers=[stats_transformer])

from rastervision.core.data import SemanticSegmentationLabelStore, PolygonVectorOutputConfig
label_store = SemanticSegmentationLabelStore(
    uri=out_dir,
    class_config=class_config,
    bbox=multi_rs_26_normalized.bbox,
    crs_transformer=multi_rs_26_normalized.crs_transformer,
    vector_outputs=[PolygonVectorOutputConfig(class_id=1)],
    smooth_output=True,
    save_as_rgb=False,
)

from rastervision.core.data import Scene

scene = Scene(
    id='beirut',
    raster_source=multi_rs_26_normalized,
    label_store=label_store)

from rastervision.core.rv_pipeline import SemanticSegmentationPredictOptions

pred_opts = SemanticSegmentationPredictOptions(
    chip_sz=512, stride=256, crop_sz='auto', batch_sz=8)

from rastervision.pytorch_learner.utils.prediction import predict_scene_ss
labels = predict_scene_ss(learner, scene, pred_opts)

scene.label_store.save(labels)

While it creates the labels.tif it's just a single black block in QGIS.

I feel like I'm missing some obvious, easy way of using the model in inference mode.

[AdeelH]

What does the scores.tif file look like? Is the chip_sz you're using for prediction the same as the one used for training? The notebook uses randomly sized chips between 50 and 100 pixels for training. If you haven't changed that value, I recommend trying chip_sz=100.

For further debugging, you may try plotting a sample chip from the raster source to see if that looks correct.

---

For the pipeline mode I run into errors immediately when trying to call rastervision pipeline with the model-bundle parameter and an infile and an outfile. (I even gdal_merged the 13 bands to a single geotiff to try this.) It always complains that "Could not read /opt/data/tmp/tmp_jr8nt_p/bundle/pipeline-config.json" or something similar.

I assume you're trying to use the rastervision predict CLI command for this. That requires you to use the model bundle created by the bundle stage and not the model bundle created by the train stage (confusing, I know). You can run the bundle stage like so:

rastervision run inprocess "config2.py" bundle -a out_dir "blog_1/"

This will create a "blog_1/bundle/model-bundle.zip" file. This is what needs to be passed to the predict command.

Also, as you realized, it's not possible to use this command with a MultiRasterSource, so you will have to pre-create a 26-band GeoTIFF to use this. Luckily, there is an alternative predict_scene command which takes a SceneConfig. Example usage:

scene_config = SceneConfig(
  raster_source=MultiRasterSourceConfig(....),
  label_store=SemanticSegmentationLabelStoreConfig(...),
)
pred_opts = SemanticSegmentationPredictOptions(...)
scene_config.to_file('scene_config.json')
pred_opts.to_file('pred_opts.json')

rastervision predict_scene --predict_options_uri "pred_opts.json" "blog_1/bundle/model-bundle.zip" "scene_config.json"

To do this in a notebook or script instead of the CLI:

from rastervision.core.predictor import ScenePredictor

scene_config = Scene(
  raster_source=MultiRasterSource(....),
  label_store=SemanticSegmentationLabelStore(...),
)
pred_opts = SemanticSegmentationPredictOptions(...)

predictor = ScenePredictor("blog_1/bundle/model-bundle.zip", pred_opts)
predictor.predict_scene(scene)

LMK if you run into any problems.

[MarkoTukiainen]

Thanks, that cleared thing up a ton. And also using that last example I was able to make some predictions and save them successfully to a tif file, though they were not exactly the same as the ones created using the

rastervision run inprocess "config.py" predict eval -a out_dir "blog/1"

method, but pretty close. I did change the chip_sz to 100 as you suggested, that might have had an impact. Nevertheless, I'm very pleased I got a concrete result out of it, the quality of the predictions isn't really all that important at this point.

One thing I forgot to ask previously, I'm running this both in the docker container but also on a jupyter lab on bare macos (instead of using the jupyter lab provided by the container, I may end up just switching my notebooks over to that), and many times I run into the issue that I should have num_workers=0. But there have been many instances where I haven't had a way to add this parameter anywhere - for example the ScenePredictor doesn't support it, and it crashes when running on macos. Is there a universal way of setting this?

Also, here's the config.py I used for training. I had to get creative with some of the new syntax since the migration guides, while helpful in pointing me to the right direction, left quite a few open questions. For example in the WindowSamplingConfig I had both size=256 and size_lims=(50, 150) parameters. Also I changed batch size to 8 (as opposed to 16 in the notebook) since the training process kept running out of memory and killed and I don't really have more memory to give the container.

Also I didn't get the lambda_transforms bit working at all so I used some albumentations I found in some of the discussions.

If I made some glaring errors here I'd be very keen to find out.

from os.path import abspath, join
from pathlib import Path

from rastervision.core.data import (
    ClassConfig, DatasetConfig, SceneConfig, RasterioSourceConfig,
    MultiRasterSourceConfig, SemanticSegmentationLabelSourceConfig,
    ReclassTransformerConfig, StatsTransformerConfig)
from rastervision.core.rv_pipeline import (SemanticSegmentationConfig, SemanticSegmentationChipOptions, WindowSamplingConfig, WindowSamplingMethod, SemanticSegmentationPredictOptions)
from rastervision.pytorch_backend import PyTorchSemanticSegmentationConfig
from rastervision.pytorch_learner import (
    SemanticSegmentationGeoDataConfig, SemanticSegmentationModelConfig, ExternalModuleConfig,
    SolverConfig, PlotOptions)

from rastervision.pytorch_learner.utils import serialize_albumentation_transform

import albumentations as A

img_dir = Path('./oscd/images')
train_labels_dir = Path('./oscd/trainlabels/')

with open(img_dir / 'train.txt', 'r') as f:
    train_val_cities = f.read().strip().split(',')

# roughly a 75-25 split: 10 train scenes and 4 validation scenes
train_cities = train_val_cities[:10]
val_cities = train_val_cities[10:]


def get_config(runner, out_dir: str) -> SemanticSegmentationConfig:
    ######################################
    # Configure scenes and training split
    ######################################
    class_cfg = ClassConfig(
        names=['no change', 'change'],
        colors=['lightgray', 'red'],
        null_class='no change')

    train_scenes = [make_scene_cfg(city) for city in train_cities]
    val_scenes = [make_scene_cfg(city) for city in val_cities]

    dataset_cfg = DatasetConfig(
        class_config=class_cfg,
        train_scenes=train_scenes,
        validation_scenes=val_scenes)

    ########################
    # Configure GeoDatasets
    ########################
    chip_options = SemanticSegmentationChipOptions(
        sampling=WindowSamplingConfig(
            method=WindowSamplingMethod.random, size=256, size_lims=(50, 150), max_windows=100))

    #from lt import lambda_transforms
    #aug_tf = serialize_albumentation_transform(
    #    tf=lambda_transforms['swap'],
    #    lambda_transforms_path=abspath('./lt.py'),
    #    dst_dir=abspath(join(out_dir)))

    transforms = A.Compose([A.Flip(), A.Transpose(), A.RandomRotate90()])
    aug_transform = A.to_dict(transforms)

    data_cfg = SemanticSegmentationGeoDataConfig(
        scene_dataset=dataset_cfg,
        sampling=chip_options.sampling,
        img_channels=26,
        img_sz=256,
        num_workers=4,
        aug_transform=aug_transform,
        plot_options=PlotOptions(
            # RV plots some sample chips and predictions during training.
            # Here we tell it to break up the image into 2 images: one with
            # RGB bands from the "before" image and one with RGB bands from the
            # "after" image.
            channel_display_groups={
                'RGB_1': [3, 2, 1],  # RGB bands from the "before" image
                'RGB_2': [16, 15, 14],  # RGB bands from the "after" image
            }))

    ########################
    # Configure the Learner
    ########################
    #model_cfg = SemanticSegmentationModelConfig()  # go with the defaults
    model_cfg = SemanticSegmentationModelConfig(
        external_def=ExternalModuleConfig(
            github_repo='AdeelH/pytorch-fpn:0.2',
            name='fpn',
            entrypoint='make_fpn_resnet',
            entrypoint_kwargs={
                'name': 'resnet50',
                'fpn_type': 'panoptic',
                'num_classes': 2,
                'fpn_channels': 256,
                'in_channels': 26,
                'out_size': (256, 256)
            }))
    solver_cfg = SolverConfig(
        lr=0.01, 
        num_epochs=10, 
        batch_sz=8, 
        one_cycle=True, 
        class_loss_weights=[1., 10.])

    #########################
    # Configure the Pipeline
    #########################

    # the Backend interfaces between the Learner and the Pipeline
    backend_cfg = PyTorchSemanticSegmentationConfig(
        data=data_cfg,
        model=model_cfg,
        solver=solver_cfg,
        log_tensorboard=False)

    # the semantic segmentation pipeline
    pipeline_cfg = SemanticSegmentationConfig(
        root_uri=out_dir, dataset=dataset_cfg, backend=backend_cfg, chip_options=chip_options, predict_options=SemanticSegmentationPredictOptions(chip_sz=256))

    return pipeline_cfg


def make_scene_cfg(city: str) -> SceneConfig:
    scene_cfg = SceneConfig(
        id=city,
        raster_source=make_raster_source_cfg(city),
        label_source=make_label_source_cfg(city))
    return scene_cfg


def make_raster_source_cfg(city: str) -> MultiRasterSourceConfig:
    uris_1 = sorted((img_dir / city / 'imgs_1/').glob('*.tif'))
    raster_source_cfgs_1 = [
        RasterioSourceConfig(uris=[str(uri)]) for uri in uris_1
    ]

    uris_2 = sorted((img_dir / city / 'imgs_2/').glob('*.tif'))
    raster_source_cfgs_2 = [
        RasterioSourceConfig(uris=[str(uri)]) for uri in uris_2
    ]

    multi_raster_source_cfg = MultiRasterSourceConfig(
        raster_sources=raster_source_cfgs_1 + raster_source_cfgs_2,
        primary_source_idx=1,
        transformers=[StatsTransformerConfig()])
    return multi_raster_source_cfg


def make_label_source_cfg(city: str) -> SemanticSegmentationLabelSourceConfig:
    uri = str(train_labels_dir / city / f'cm/{city}-cm.tif')
    raster_source_cfg = RasterioSourceConfig(
        uris=[uri],
        transformers=[
            ReclassTransformerConfig(mapping={
                1: 0,  # "no change": 1 --> 0
                2: 1  # "change": 2 --> 1
            })
        ])
    label_source_cfg = SemanticSegmentationLabelSourceConfig(
        raster_source=raster_source_cfg)
    return label_source_cfg

[AdeelH]

and many times I run into the issue that I should have num_workers=0. But there have been many instances where I haven't had a way to add this parameter anywhere - for example the ScenePredictor doesn't support it, and it crashes when running on macos. Is there a universal way of setting this?

You can set the environment variable RASTERVISION_PREDICT_NUM_WORKERS=0.

Recommend chaning github_repo='AdeelH/pytorch-fpn:0.2' to github_repo='AdeelH/pytorch-fpn:0.3'.

Can you share what errors you run into with lambda_transforms?

[AdeelH]

@MarkoTukiainen Let me also take this opportunity to plug a Geospatial Machine Learning with Cloud Services workshop that I will be conducting next month (May 15). The workshop is fully virtual and will heavily focus on using Raster Vision. Please consider registering if this is something that could be useful for you.

[Nizar150]

Hi marko could you please share the modified notebook for change detection? Here is my mal nizarelharrouni@gmail.com .
Thank you

[MarkoTukiainen]

https://colab.research.google.com/drive/1lZiyZhjgiLMK29Hk5rIBQv1TEL6_WNjV

[Nizar150]

Thank you marko are your modification to migrate it to the actual version of the rastervion in it?

[MarkoTukiainen]

I didn't really create a publishable notebook of my changes, however the config.py posted above should be fully working. Especially if you amend it with the recommended fpn version change.

[Nizar150]

But you did implement the changes recommended by adeel didn't you?.as for the second part i did not understand .
I just want to try it and then train on some custom data of mine.
Thank you