Testing new implementation

[cyriltasse]

Hi-

Ok - so I've been implementing various changed to DynSpecMS. Branch: MTilde_ASync

• First the bug of the off being disrtibuted it the top left quadrant is fixed
• the off are distributed in all facets with the same number per facet (and a minimum number of --nMinOffPerFacet (default 5))
• the Jones are not anymore applied at the inverse but as the Hermitian transpose as described in the DDFacet paper (this is the proper way to do). Just dealing with scalar case so far, but I'll have to extend it to full pol if we redo the calibration with the constrained solvers since we use diagonal solve in one step.
• DynSpecMS is now asyncronous, meaning it prepares the next datachunk as it processes the currect one. Make it more efficient by about 25%

The two first ones will highly improve the statistics robustness, and potential to detect and reject stuff. The last step should improve the results to get better spectra. In the few tests I did I see some improvement but nothing dramatic.

This is what the off/on (blue/green) now looks like. Only facets covered by the search radius and concerned with offs beeing scattered accross them.

This is the comparison between before (left) and now (right):

Noise is lower by 2% in the newer version with lower artifacts (in a few places it's higher, I'll experiment quickly with clipping the data with the high gains, but I think it's not too bad given we now will have good off statistics).

Joe - time to add you proper motion correction? Give me a module and I can update the code

The command line I've used was:

ms2dynspec.py --ms big-mslist.txt --data DATA --model DDF_PREDICT --sols [DDS3_full_smoothed,DDS3_full_slow_merged] --rad 2. --SolsDir SOLSDIR --BeamModel LOFAR --BeamNBand 1 --DicoFacet image_full_ampphase_di_m.NS3.DicoFacet --noff 100

The facet layout is given by --DicoFacet argument

[cyriltasse]

We need to do tests on one of the nodes the data would be reduced on, but if DynSpecMS turns out to be too slow, I can probably accelerate it...

[PhilippeZarka]

Impressive.
If I can help with the input catalog or anything...

[jrcallingham]

Nice Cyril. What is the radius of the large green circle? 2.2 deg? We might want to make it a little larger since some stars are detected ~3 deg from the pointing centre but will not have their dynamic spectrum produced.

[jrcallingham]

As I said in email, I will get you a modified target list and proper motion code tonight/tomorrow

[cyriltasse]

Yeah - 2 degrees that's what we have previously use. We could go to 3 but computing might be x2-3. Let's see what we can afford after we have tested and quantified computing time?

[cyriltasse]

Hey,

On the left panel is the right hand image of the above and when you apply to Jones clipping, you get the right panel image bellow

It looks pretty good, it could really improve things...

[PhilippeZarka]

Promising...

[jrcallingham]

This is looking fantastic Cyril! This kind of "scruff" really limits some analysis and was producing false positives. However, I still dont understand what produces the "scruff" at the top right then? If the Jones clipping got rid of the stuff in the middle, why not that? Maybe it is lower SNR?

[cyriltasse]

However, I still dont understand what produces the "scruff" at the top right then? If the Jones clipping got rid of the stuff in the middle, why not that? Maybe it is lower SNR?

Good question, I don't really know. But clipping don't mean you're left with good solutions. Just less extreme. And there is also the visibilities themselves that can go crazy (we have applied the DI Jones as the inverse (as the whole rest of the world) - and not as the Hermitian transpose, so low amplitudes blow up the noise). And there is the facts that some time-freq has less (flagged) visibilities stacked - therefore more noisy average. In my process, this is taken into account, I reduce the spectra taking the weights into account.

It will be important to test on other fields before we decide whether we want to use Jones clipping (but I think we will want to). I want to test on fields that has very bad facets for instance.

[cyriltasse]

I'm now editing this reprocessing script as suggested by Tim. I hope we can run a first test with a few tens of fields by next week. I'd like to launch the reprocessing before christmass.

[jrcallingham]

Hey mate,

Working on it. Probably will not get it to you today but will definitely get it to you Monday afternoon.

[cyriltasse]

@jrcallingham would be cool - then the first batch of tests can be run using the pm correction.

[jrcallingham]

Hey Cyril - as I said in the general email to the group, here is the proper motion code. I dont want to implement it directly as you know you code best but I have assumed you have already read in the table at some point (see comments etc) ``

# Proper motion correction

from astropy.io import fits
from astropy.coordinates import SkyCoord, Distance
from astropy.time import Time
import astropy.units as u

# Assuming two things, you have read in the table and made the vairables ra, dec, pmra etc, and you the date of the observation (date_obs) in YYYY-MM-DD format.

# In terms of a pipeline I would have an if statement here that if pmra is a real value, then do the following, 
# otherwise just extract at the reported ra,dec position

c = SkyCoord(ra=ra * u.deg,
            dec=dec * u.deg,
            distance=Distance(parallax=parallax * u.mas,allow_negative=True),
            pm_ra_cosdec=pmra * u.mas/u.yr,
            pm_dec=pmdec * u.mas/u.yr,
            obstime=Time(ref_epoch, format='decimalyear'))

c_lotss = SkyCoord(ra=ra_lotss * u.deg,
            dec=dec_lotss * u.deg,
            obstime=Time(date_obs, format='iso'))

epoch_lotss = Time(date_obs, format='iso')

print('Performing proper motion corrections for '+field)

c_gaia_to_lotss_epoch = c.apply_space_motion(epoch_lotss)

# Then to extract the correct ra and dec just do it at  c_gaia_to_lotss_epoch.ra and c_gaia_to_lotss_epoch.dec```

[jrcallingham]

Note that ra_lotss and dec_lotss are just the centre of the field. You can probably not set them too

[cyriltasse]

@jrcallingham Cool - thanks!

@mhardcastle @tim

I see some weird things happening with DDF/kMS reprocessing the data... I'm working with P236+53/L470106

Looks like the DicoModel was made with "--Beam-At facet", but when I do that the resid look ugly, si I tried using "--Beam-At tessel" and now they look good. Also the option is not present in the kMS parset at the time (was tessel by default)... DDFacet log says version "dev-c563ecb". I tried to git checkout to investigate but it doesn't match with a tag/hash.

Do you have any clues what's going on? How I could dig in the issue?