Planetary Computer Hub downtime for Saturday, August 20th [completed]

[TomAugspurger]

The migration is finished and the Hub is up and running on the new AKS cluster at the same URL: https://pccompute.westeurope.cloudapp.azure.com/

Reach out here or email planetarycomputer@microsoft.com if you run into any issues after the migration.

---

The Planetary Computer Hub will be down for some maintenance on Saturday, August 20th. During this time, you won’t be able to log into the Hub, start notebook servers, or connect to the Dask Gateway. The data and APIs hosted by the Planetary Computer and APIs will not be affected by the downtime. The rest of the Planetary Computer, including the data, APIs, Explorer, and all of the other ways to compute on data hosted by the Planetary Computer will not be affected.

We expect the Hub to be down for about 8 hours. We'll post updates here during the downtime and once things are up and running again.

Why are we doing this?

The Planetary Computer Hub is a JupyterHub deployed on AKS. When I initially deployed the AKS cluster, I failed to set a few flags that will be handy for future development and maintenance of the Hub and AKS cluster. Unfortunately, these settings can’t be changed on an existing AKS cluster, so we need to create a new one.

The JupyterHub deployment has a few critical pieces of state that need to be migrated between the cluster. To avoid any data loss in your home directories (specifically, changes made between the time the new cluster is ready and the old cluster is shut down), we’re going to scale down the old Hub and stop all the notebook servers before bringing up the new Hub and transferring all of the state to the new cluster.

---

Status updates:

• 2022-08-20 06:11: Starting the maintenance. The old Hub is shutting down.
• 2022-08-20 06:18: Snapshots started. This will take a while.
• 2022-08-20 09:15: Restore started. This will take even longer.
• 2022-08-20 11:40: Restore finished.
• 2022-08-20 11:35: DNS migration started.
• 2022-08-20 11:41: DNS migration finished.
• 2022-08-20 11:55: Migration finished

[weiji14]

Hi there, just wondering if the kbatch service at https://pccompute.westeurope.cloudapp.azure.com/compute/services/kbatch/ is currently down? I've been trying to submit a batch job following https://kbatch.readthedocs.io/en/latest/examples/shell-script.html but it's returning this HTTP 500 Internal Server error:

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/bin/kbatch", line 8, in <module>
    sys.exit(cli())
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/kbatch/cli.py", line 146, in submit
    result = _core.submit_job(
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/kbatch/_core.py", line 219, in submit_job
    r.raise_for_status()
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/httpx/_models.py", line 736, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'https://pccompute.westeurope.cloudapp.azure.com/compute/services/kbatch/jobs/'
For more information check: https://httpstatuses.com/500

The above is using kbatch==0.3.2. I also tried using kbatch==0.4.1 but the traceback was a bit weird:

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/bin/kbatch", line 8, in <module>
    sys.exit(cli())
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/kbatch/cli.py", line 281, in submit_job
    result = _core.submit_job(
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/kbatch/_core.py", line 152, in submit_job
    data = make_job(job, profile=profile).to_dict()
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/kbatch/_backend.py", line 240, in make_job
    job_spec = _make_job_spec(job, profile, labels, annotations)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/kbatch/_backend.py", line 87, in _make_job_spec
    resources = profile.get("resources", {})
AttributeError: 'str' object has no attribute 'get'

Also tried running a simple command like kbatch job list using kbatch==0.3.2 but am getting a similar '500 Internal Server Error', so am wondering if it's just that the server is currently offline or something.

[weiji14]

No worries, I'll let it sit in the queue for a while. It's just a batch job anyway so not in a rush. Thanks though for the quick troubleshooting, really appreciate it!

[TomAugspurger]

@weiji14 it seems like your Pod job was scheduled to a node, but is having trouble pulling the image:

Normal  BackOff  3m (x12131 over 45h)  kubelet  Back-off pulling image "mcr.microsoft.com/planetary-computer/gpu-pytorch:2022.8.1.0"

We noticed some issues yesterday with a separate service where the node was failing to pull container images. I might kill that pod (and maybe node) to see if it fixes the problem. You'll need to resubmit your job.

[weiji14]

No problem, feel free to kill any jobs/pods/nodes. I actually tried doing kbatch job delete <job-name> but it still says 'pending' for some reason. Maybe only running/completed jobs can be deleted?

[TomAugspurger]

I'm not sure...

I did just notice though: the tag gpu-pytorch:2022.8.1.0 isn't valid. We need a better system for documenting this, but for gpu-pytorch they're at https://mcr.microsoft.com/v2/planetary-computer/gpu-pytorch/tags/list.

Can you try with the image mcr.microsoft.com/planetary-computer/gpu-pytorch:2022.05.2.0?

[weiji14]

I did just notice though: the tag gpu-pytorch:2022.8.1.0 isn't valid. We need a better system for documenting this, but for gpu-pytorch they're at https://mcr.microsoft.com/v2/planetary-computer/gpu-pytorch/tags/list.

Can you try with the image mcr.microsoft.com/planetary-computer/gpu-pytorch:2022.05.2.0?

Yep, changing the image did the trick! I was just using the tags at https://github.com/microsoft/planetary-computer-containers/tags and assuming they corresponded to the docker image tags...

Anyways, the batch job is finally running now, and I can see the logs and all. Thanks again for the quick troubleshooting!