Dijittered HiRISE cubes are not recognized by ALE

[thareUSGS]

For HiRISE images that have jitter, UArizona has a processing method to dejitter the HiRISE using the individual CCDs. Once this step is completed, an ISIS cube is re-created and SPICE is added back in.

I don't see anything obvious in the created cube (label) to cause ALE to fail (or to not recognize the instrument). Running isd_generate.py (to create an ISD for GXP, ASP, or ISIS), I get this error:

python isd_generate.py ESP_060697_1985_RED_DEJITTERED.8bit.cub
Reading: ESP_060697_1985_RED_DEJITTERED.8bit.cub
File ESP_060697_1985_RED_DEJITTERED.8bit.cub: No Such Driver for Label

I also don't see any special handling for this style of HiRISE for the original ISIS (Socet SET) code, socetlinescankeywords. Note the example linked works fine in socetlinescankeywords.

Image for ALE testing ~1 GB):
ESP_060697_1985_RED_DEJITTERED.8bit.cub.gz
Currently, there are limited permissions to download this file. DOI personnel can download it.

[thareUSGS]

Using ALE verbose reporting states many sections are missing from the Dejittered cube (SunPosition, BodyRotation, InstrumentPosition, NaifKeywords). As a “noproj” cube (aka “ideal” camera-geometry), these sections are likely no longer needed, but still apparently required by ALE.

[thareUSGS]

In consultation with Ross, who implemented hiproc (dejitter implementation in Python), I will summarize his response to this issue.

1. Regarding the dejitter methods used by the HiRISE Team updated the pointing (SPICE), Ross stated, "I believe that hijitter is essentially just moving pixels around in the focal plane, but not really altering the gross position and pose of the original image."

That implies we are likely safe using ALE on a typically derived (non-dejittered) HiRISE image to generate the ISD (pointing metadata) for the HiRISE dejittered image. This is likely a probable workaround.

2. Perhaps there could be a stab to implement an ALE driver to support “noproj” cubes. Note, these dejittered cubes do contain a "Kernel" group which is enough to run campt. That hopefully implies an ALE driver could be written.

[Kelvinrr]

The thing to do then is to use a generic ideal driver with ISISLabel + NaifSpice. This shouldn't be too hard if there exists a tell for a noproj cube, that way it won't be picked up by other drivers.

[acpaquette]

@thareUSGS I have been meaning to respond to this since investigating it in April. The main issue with the dejittered cubes is that they contain an InstrumentPointing table but no other spice tables or NaifKeywords group.

From inspecting the history of the cube, I think I can make a guess on the issue. During the process there is a specific step where the spice tables are not attached to the cube, only the kernels are (attach=NO):

spiceinit from=./matchMaster.cub web=false attach=NO cksmithed=FALSE ckrecon=TRUE ckpredicted=FALSE cknadir=FALSE spksmithed=FALSE spkrecon=TRUE spkpredicted=FALSE shape=SYSTEM startpad=0.0 endpad=0.0 url=https://services.isis.astrogeology.usgs.gov/cgi-bin/spiceinit.cgi port=443

I am not sure if this is necessary or just a space saver. From here, when no proj is called, only the kernels are translated with the cube rather than all of the spice tables. This results in an IdealCamera with no tables or NaifKeywords group. The ISIS ideal camera can handle this and will create the necessary spice information to create a camera model. There will be a longer initial spin up time to any application as you will be reading information directly from the kernels each time to create the model.

In contrast ALE doesn't support this and cannot create an ISD from a cube with only a kernels group. ALE could support this but I am not sure what is easier/more correct. It would be nice to fix all of the dejitter HiRISE cubes to have the necessary spice information to process under the current implementation of ALE, but it would likely be easier to just add a driver to support the instrument as is.

Let me know what would be the best way forward or if we can chat with the HiRISE team to come to some resolution/understanding

[rbeyer]

My apologies for not notcing this one before now. Are you saying that this error would go away if during the "spiceinit" call that makes the products, we simply specified "attach=yes"?

I actually can't find any place in either the HiRISE production pipelines or hiproc where "attach=NO" is specified, and the default for spiceinit is for "attach" to be TRUE, so I'm not sure where this is happening. Unless this is happening internal to the ISIS noproj command somehow?

[acpaquette]

@rbeyer Correct, from the image Trent posted above, this is the result from running cathist with mode=brief

hi2isis from=/HiRISE/Data/EDRgen/Version_001/ESP/ORB_060600_060699/ESP_060697_1985/-
       ESP_060697_1985_RED5_0.IMG to=/HiRISE/Data/EDR_Stats/ESP/ORB_060600_060699/ESP_060697_1985/ESP_060697_-
     1985_RED5_0.cub lsbgap=true unlut=true
hical from=/HiRISE/Data/EDR_Stats/ESP/ORB_060600_060699/ESP_060697_1985/ESP_06069-
       7_1985_RED5_0.cub conf=/HiRISE/Configuration/HiCal/hical.pipelines.conf to=/tmp/ESP_060697_1985_RED5_0.cnode03:7303.hical.cub+SignedWord+0.0:1.5 units=IOF opath=/HiRISE/Data/HiCal/ESP/ORB_060600_060699/ESP_060697_1985
fx f1=/tmp/ESP_060697_1985_RED5_0.cnode03:7303.hical.cub to=/tmp/Fxtemp.23970.fx.cub equation="((F1/(0.99369399999999996620+(0.00000074566900000000*line)+(--0.0-
            0000000002204540000*line*line)))*(line<16829.0) +
            (F1*(line>=16829.0)))" mode=CUBES
specadd from=/tmp/Fxtemp.23970.fx.cub match=/tmp/ESP_060697_1985_RED5_0.cnode03:7303.hical.cub to=/tmp/ESP_060697_1985_RED5_0.cnode03:7303.fx.cub
cubenorm from=/tmp/ESP_060697_1985_RED5_0.cnode03:7303.fx.cub to=/tmp/ESP_060697_1985_RED5_0.cnode03:7303.cubenorm.cub+SignedWord+0.0:1.5 format=TABLE statsource=TABLE fromstats=/tmp/ESP_060697_1985_RED5_0.cnode03:7303.cubenorm_fix.tab direction=COLUMN normalizer=AVERAGE mode=DIVIDE preserve=FALSE
hidestripe from=/tmp/ESP_060697_1985_RED5_0.cnode03:7303.cubenorm.cub to=/tmp/ESP_060697_1985_RED5_0.cnode03:7303.hidestripe.cub+SignedWord+0.0:1-
     .5 correction=ADD parity=EVEN
hidestripe from=/tmp/ESP_060697_1985_RED5_0.cnode03:7303.hidestripe.cub to=/HiRISE/Data/HiCal/ESP/ORB_060600_060699/ESP_060697_1985/ESP_060697_1985-
     _RED5_0.temp.cub+SignedWord+0.0:1.5 correction=ADD parity=ODD
histitch from1=/HiRISE/Data/HiCal/ESP/ORB_060600_060699/ESP_060697_1985/ESP_060697_1-
        985_RED5_0.hical.cub from2=/HiRISE/Data/HiCal/ESP/ORB_060600_060699/ESP_060697_1985/ESP_060697_1-
        985_RED5_1.hical.cub to=/HiRISE/Data/HiStitch/ESP/ORB_060600_060699/ESP_060697_1985/ESP_060697_1-
     985_RED5.temp.cub balance=TRUE skip=7 seamsize=11 channel=0
algebra from=/HiRISE/Data/HiStitch/ESP/ORB_060600_060699/ESP_060697_1985/ESP_060697-
       _1985_RED5.histitch.cub to=/HiRISE/Data/HiStitch/ESP/ORB_060600_060699/ESP_060697_1985/ESP_060697_1-
     985_RED5.balance.cub+SignedWord+0.0:1.5 operator=UNARY a=1 c=0
editlab from=/HiRISE/Data/HiStitch/ESP/ORB_060600_060699/ESP_060697_1985/ESP_060697-
       _1985_RED5.balance.cub options=ADDKEY grpname=Instrument keyword=Special_Processing_Flag value=NOMINAL
spiceinit from=/HiRISE/Data/HiStitch/ESP/ORB_060600_060699/ESP_060697_1985/ESP_060697-
       _1985_RED5.balance.cub web=false attach=TRUE cksmithed=TRUE ckrecon=TRUE ckpredicted=FALSE cknadir=FALSE spksmithed=TRUE spkrecon=TRUE spkpredicted=FALSE shape=USER model=/HiRISE/Data/NAIF/molaDTM/molaMarsPlanetaryRadius_HiRISE_v2.cub startpad=0.0 endpad=0.0 url=http://services.isis.astrogeology.usgs.gov/cgi-bin/spiceinit.cgi port=80
spiceinit from=/tmp/ESP_060697_1985_pnode01:18177/ESP_060697_1985_RED5.balance.cub web=false attach=TRUE cksmithed=TRUE ckrecon=TRUE ckpredicted=FALSE cknadir=FALSE spksmithed=TRUE spkrecon=TRUE spkpredicted=FALSE shape=SYSTEM startpad=0.0 endpad=0.0 url=https://services.isis.astrogeology.usgs.gov/cgi-bin/spiceinit.cgi port=443
cubeatt from=ESP_060697_1985_RED5.balance.cub to=./matchMaster.cub proptables=true
spiceinit from=./matchMaster.cub web=false attach=NO cksmithed=FALSE ckrecon=TRUE ckpredicted=FALSE cknadir=FALSE spksmithed=FALSE spkrecon=TRUE spkpredicted=FALSE shape=SYSTEM startpad=0.0 endpad=0.0 url=https://services.isis.astrogeology.usgs.gov/cgi-bin/spiceinit.cgi port=443
noproj from=./ESP_060697_1985_RED8.dejittered.copy.FROM9.cub match=./matchMaster.cub to=ESP_060697_1985_RED8.dejittered.cub source=FROMUSER summingmode=1.0 interp=CUBICCONVOLUTION sampexp=0.0 lineexp=0.0
cam2cam from=./ESP_060697_1985_RED8.dejittered.copy.FROM9.cub match=match.cub to=ESP_060697_1985_RED8.dejittered.cub interp=CUBICCONVOLUTION
editlab from=ESP_060697_1985_RED8.dejittered.cub options=SETKEY grpname=Instrument keyword=ImageJitterCorrected value=1
handmos from=/tmp/ESP_060697_1985_pnode01:18177/ESP_060697_1985_RED8.dejittered.cub mosaic=ESP_060697_1985_RED.NOPROJ.cub priority=beneath insample=1 inline=1 inband=1 outsample=2 outline=2 outband=1 matchbandbin=FALSE matchdem=FALSE create=NO
editlab from=ESP_060697_1985_RED.NOPROJ.cub options=modkey grpname=Archive keyword=ProductId value=ESP_060697_1985_RED
editlab from=ESP_060697_1985_RED.NOPROJ.cub options=modkey grpname=Instrument keyword=MatchedCube value=ESP_060697_1985_RED5
editlab from=ESP_060697_1985_RED.NOPROJ.cub options=ADDKEY grpname=Archive keyword=SourceProductId value=(ESP_060697_1985_RED0_0, ESP_060697_1985_RED0_1,
         ESP_060697_1985_RED1_0, ESP_060697_1985_RED1_1,
         ESP_060697_1985_RED2_0, ESP_060697_1985_RED2_1,
         ESP_060697_1985_RED3_0, ESP_060697_1985_RED3_1,
         ESP_060697_1985_RED4_0, ESP_060697_1985_RED4_1,
         ESP_060697_1985_RED5_0, ESP_060697_1985_RED5_1,
         ESP_060697_1985_RED6_0, ESP_060697_1985_RED6_1,
         ESP_060697_1985_RED7_0, ESP_060697_1985_RED7_1,
         ESP_060697_1985_RED8_0, ESP_060697_1985_RED8_1)
editlab from=ESP_060697_1985_RED.NOPROJ.cub options=SETKEY grpname=Kernels keyword=InstrumentPointing value=(Table, $mro/kernels/ck/mro_sc_psp_190709_190715.bc,
         $mro/kernels/fk/mro_v16.tf)
editlab from=ESP_060697_1985_RED.NOPROJ.cub options=SETKEY grpname=Kernels keyword=InstrumentPosition value=$mro/kernels/spk/mro_psp52.bsp
stretch from=ESP_060697_1985_RED.NOPROJ.cub to=ESP_060697_1985_RED_DEJITTERED.8bit.cub+8bit+1:254 usepercentages=False readfile=False pairs="0:0 0.08615058227384:1 0.143385888063:254" lis=1.0 lrs=1.0 his=254 hrs=254

These particular commands seem to be the issue, all three of which happen before no proj:

spiceinit from=/tmp/ESP_060697_1985_pnode01:18177/ESP_060697_1985_RED5.balance.cub web=false attach=TRUE cksmithed=TRUE ckrecon=TRUE ckpredicted=FALSE cknadir=FALSE spksmithed=TRUE spkrecon=TRUE spkpredicted=FALSE shape=SYSTEM startpad=0.0 endpad=0.0 url=https://services.isis.astrogeology.usgs.gov/cgi-bin/spiceinit.cgi port=443
cubeatt from=ESP_060697_1985_RED5.balance.cub to=./matchMaster.cub proptables=true
spiceinit from=./matchMaster.cub web=false attach=NO cksmithed=FALSE ckrecon=TRUE ckpredicted=FALSE cknadir=FALSE spksmithed=FALSE spkrecon=TRUE spkpredicted=FALSE shape=SYSTEM startpad=0.0 endpad=0.0 url=https://services.isis.astrogeology.usgs.gov/cgi-bin/spiceinit.cgi port=443

We spiceinit the hirise image, then copy it into a new cube called matchMaster.cub, then run spiceinit again with attach=NO. I don't know if the pipeline has changed since this cube was processed. Looks like it was processed around April 25th, 2023