killMS is getting stuck #19
Description
Hi all,
I am truing to calibrate an image that has a very strong source at the center (3C273). But killMS gets stuck at the first step. Can you please point me if there is a way around or how to identify the issue. Relevant portion of .err and .out are as follows. This is very similar to the issue #16 .
- 01:00:49 - killMS | In batch mode, running killMS on the following MS:
- 01:00:49 - killMS | /myPATH/1688399183_cal_avg_data-3C273-corr.MS
- 01:00:49 - killMS | /myPATH/1688399183_cal_avg_data-3C273_u0_8-corr.MS
- 01:00:49 - killMS | /myPATH/1688399183_cal_avg_data-3C273_d0_8-corr.MS
- 01:00:49 - killMS | /myPATH/1688399183_cal_avg_data-3C273_l0_8-corr.MS
- 01:00:49 - killMS | /myPATH/1688399183_cal_avg_data-3C273_r0_8-corr.MS
- 01:00:49 - killMS | Running kMS.py BatchCurrentParset.parset --MSName=/myPATH/1688399183_cal_avg_data-3C273-corr.MS
- 01:00:50 - killMS | �[1m�[91mSuccessfully read BatchCurrentParset.parset parset�[0m�[0m
- 01:00:50 - killMS | Checking system configuration:
- 01:00:50 - killMS | Max shared memory size is 90% of total RAM size
- 01:00:50 - killMS | sysctl vm.max_map_count = 4124832
- 01:00:50 - AsyncProcessPool | �[1m�[92mFixing parent process to vthread 0�[0m�[0m
- 01:00:50 - AsyncProcessPool | Worker affinities not specified, leaving unset
- 01:00:50 - ClassFITSBeam | using standard Cattery.Siamese module from /usr/local/lib/python3.9/dist-packages/Cattery/Siamese
- 01:00:50 - AsyncProcessPool | �[1m�[92mFixing parent process to vthread 0�[0m�[0m
- 01:00:50 - AsyncProcessPool | Worker affinities not specified, leaving unset
- 01:00:50 - killMS | �[1m�[91mPredict Mode: using column DDF_PREDICT�[0m�[0m
- 01:00:50 - ClassVisServer | Center of frequency domains [MHz]: [815.93359375]
- 01:00:50 - ClassVisServer | VisToSolsChanMapping [0, 0, 0, 0, 0, 0, 0, 0, ...., 0, 0, 0, 0, 0, 0, 0, 0], (len = 2048)
- 01:00:50 - ClassVisServer | SolsToVisChanMapping [(0, 2048)], (len = 1)
- 01:00:50 - killMS | Column CORRECTED_DATA_DI not in MS
- 01:00:50 - ClassMS | Putting column CORRECTED_DATA_DI in /myPATH/1688399183_cal_avg_data-3C273-corr.MS
- 01:00:51 - ClassVisServer | Center of frequency domains [MHz]: [815.93359375]
- 01:00:51 - ClassVisServer | VisToSolsChanMapping [0, 0, 0, 0, 0, 0, 0, 0, ...., 0, 0, 0, 0, 0, 0, 0, 0], (len = 2048)
- 01:00:51 - ClassVisServer | SolsToVisChanMapping [(0, 2048)], (len = 1)
- 01:00:52 - ClassWeighting | Weighting in Natural mode
- 01:00:53 - ClassVisServer | Min weight is 1.000000 max is 1.000000
- 01:00:54 - ClassWeighting | Weighting in Natural mode
- 01:00:55 - ClassVisServer | Min weight is 1.000000 max is 1.000000
- 01:00:55 - AsyncProcessPool | waiting for worker processes to start up
- 01:00:56 - AsyncProcessPool | �[1m�[92mFixing parent process to vthread 0�[0m�[0m
- 01:00:56 - AsyncProcessPool | Worker affinities not specified, leaving unset
- 01:00:56 - AsyncProcessPool | waiting for worker processes to start up
- 01:00:56 - ClassVisServer | =================================================
- 01:00:56 - ClassVisServer | �[1m�[91mLoad vis chunk #0: start�[0m�[0m
- 01:00:56 - ClassVisServer | Reading next data chunk in [ 0.00, 2.37] hours
- 01:00:56 - ClassMS | [/myPATH/1688399183_cal_avg_data-3C273-corr.MS] Reading rows [0 -> 84180]
- 01:00:58 - ClassMS | data = CORRECTED_DATA
- 01:01:15 - ClassVisServer | Channels are equidistant, can go fast
- 01:01:24 - ClassVisServer | Estimating PreApply directions at the pointing center
- 01:01:24 - ClassVisServer | PreApply Jones matrix has 1 directions
- 01:01:24 - ClassVisServer | �[1m�[91mLoad vis chunk #0: done�[0m�[0m
- 01:01:24 - ClassVisServer | =================================================
- 01:01:24 - ClassVisServer | �[1m�[91mLoad vis chunk #0: start�[0m�[0m
- 01:01:24 - ClassVisServer | Reading next data chunk in [ 0.00, 2.37] hours
- 01:01:24 - ClassMS | [/myPATH/1688399183_cal_avg_data-3C273-corr.MS] Reading rows [0 -> 84180]
- 01:01:25 - ClassMS | data = DDF_PREDICT
- 01:01:41 - ClassVisServer | Channels are equidistant, can go fast
- 01:01:51 - ClassVisServer | Estimating PreApply directions at the pointing center
- 01:01:51 - ClassVisServer | PreApply Jones matrix has 1 directions
- 01:01:51 - ClassVisServer | �[1m�[91mLoad vis chunk #0: done�[0m�[0m
- 01:01:51 - ClassWirtingerSolver | DT=8538.000000, dt=60.000000, nt=11.000000
The out file is as follows:
/usr/local/src/killMS/killMS/Parset/DefaultParset.cfg
__ _ __ __ ____ ____ ______
[ | _ (_) [ | [ | |_ \ / _|.' ____ \
| | / ] __ | | | | | \/ | | (___ \_|
| '' < [ | | | | | | |\ /| | _.____`.
| |`\ \ | | | | | | _| |_\/_| |_ | \____) |
[__| \_][___][___][___]|_____||_____| \______.'
This is version : �[1m�[91m3.0.0.2�[0m�[0m
�[1m�[91m Selected Options:�[0m�[0m
�[1m�[92m* Data-related options�[0m�[0m
- MSName ............................................ = /myPATH/1688399183_cal_avg_data-3C273-corr.MS
- ConcatFreq ........................................ = False
- TChunk ............................................ = 15
- InCol ............................................. = CORRECTED_DATA
- OutCol ............................................ = CORRECTED_DATA_DI
- FreePredictGainColName ............................ = None
- FreePredictColName ................................ = None
- Parallel .......................................... = 1
�[1m�[92m* Sky catalog related options�[0m�[0m
- SkyModel .......................................... = ''
- kills ............................................. = ''
- invert ............................................ = False
- Decorrelation ..................................... = ''
- FreeFullSub ....................................... = 0
- SkyModelCol ....................................... = DDF_PREDICT
- AppendCalSource ................................... = None
�[1m�[92m* Sky FITS Restored Cube�[0m�[0m
- FITSSkyModel ...................................... = None
- FITSNoiseBox ...................................... = 300
- FITSTh ............................................ = 20
�[1m�[92m* Beam Options�[0m�[0m
- BeamModel ......................................... = None
- BeamAt ............................................ = facet
- PhasedArrayMode ................................... = AE
- DtBeamMin ......................................... = 5
- CenterNorm ........................................ = True
- NChanBeamPerMS .................................... = 1
- FITSFile .......................................... = beam_$(corr)_$(reim).fits
- FITSParAngleIncDeg ................................ = 5
- FITSLAxis ......................................... = -X
- FITSMAxis ......................................... = Y
- FITSFeed .......................................... = ''
- FITSVerbosity ..................................... = 1
- FeedAngle ......................................... = 0
- ApplyPJones ....................................... = 0
- FlipVisibilityHands ............................... = 0
- FITSFeedSwap ...................................... = 0
- FITSFrame ......................................... = altaz
�[1m�[92m* Sky DDFacet image related options�[0m�[0m
- BaseImageName ..................................... = ''
- DicoModel ......................................... = None
- NodesFile ......................................... = None
- ImagePredictParset ................................ = ''
- OverS ............................................. = None
- MaskImage ......................................... = None
- wmax .............................................. = None
- MaxFacetSize ...................................... = None
- MinFacetSize ...................................... = None
- RemoveDDFCache .................................... = False
- DDFCacheDir ....................................... = ''
- FilterNegComp ..................................... = False
- NChanPredictPerMS ................................. = 0
- ThSolve ........................................... = 0.0
�[1m�[92m* Data Selection�[0m�[0m
- UVMinMax .......................................... = None
- ChanSlice ......................................... = None
- FlagAnts .......................................... = ''
- DistMaxToCore ..................................... = 10000.0
- FillFactor ........................................ = 1.0
- FieldID ........................................... = 0
- DDID .............................................. = 0
�[1m�[92m* Weighting scheme�[0m�[0m
- Resolution ........................................ = 0.0
- WeightInCol ....................................... = None
- WeightFlagTh ...................................... = 0
- Weighting ......................................... = Natural
- Robust ............................................ = 0.0
- WeightUVMinMax .................................... = None
- WTUV .............................................. = 1
�[1m�[92m* Action options�[0m�[0m
- DoPlot ............................................ = 0
- SubOnly ........................................... = 0
- NCPU .............................................. = 0
- DoBar ............................................. = 1
- NThread ........................................... = 1
- DebugPdb .......................................... = 1
- UpdateWeights ..................................... = 1
�[1m�[92m* PreApply killMS Solutions�[0m�[0m
- PreApplySols ...................................... = ['']
- PreApplyMode ...................................... = ['']
�[1m�[92m* Solution-related options�[0m�[0m
- ExtSols ........................................... = ''
- ClipMethod ........................................ = ['ResidAnt']
- OutSolsName ....................................... = ''
- ApplyToDir ........................................ = 0
- MergeBeamToAppliedSol ............................. = 0
- ApplyMode ......................................... = AP
- SkipExistingSols .................................. = 0
- SolsDir ........................................... = None
�[1m�[92m* Smoother solver�[0m�[0m
- SmoothEnable ...................................... = False
- SmoothInterpLevel ................................. = Inner
- SmoothInterpMode .................................. = ['Amp:Unit', 'Phase:TEC+CPhase', 'Amp:MedianFilter_5_0_7', 'Amp:Clip_0.5_1.5', 'Phase:Zero']
- AlsoSmoothATeam ................................... = 1
�[1m�[92m* Compression�[0m�[0m
- CompressionMode ................................... = None
- CompressionDirFile ................................ = None
- MergeStations ..................................... = None
�[1m�[92m* Solver options�[0m�[0m
- SolverType ........................................ = CohJones
- AsyncPredict ...................................... = True
- PrecisionDot ...................................... = D
- PolMode ........................................... = IFull
- dt ................................................ = 1.0
- NChanSols ......................................... = 1
- Sequential ........................................ = 1
�[1m�[92m* CohJones additional options�[0m�[0m
- NIterLM ........................................... = 7
- LambdaLM .......................................... = 1
- LambdaTk .......................................... = 0.0
importing InterpolatedBeams in standalone mode, ignoring MeqTrees-related stuff
�[1m�[91m MS PROPERTIES: �[0m�[0m
- File Name: �[1m�[92m/myPATH/1688399183_cal_avg_data-3C273-corr.MS�[0m�[0m
- Column Name: �[1m�[92mCORRECTED_DATA�[0m�[0m
- Selection: �[1m�[92mFIELD_ID==0 && DATA_DESC_ID==0�[0m�[0m
- Pointing center: (ra, dec)=(12:29:06.700, +02.03.08.600)
- Frequency = 544.07 MHz [ 544.07 -> 1087.80 MHz]
- Wavelength = 0.38 meters
- Time bin = 10.0 seconds
- Total Integration time = 2.37 hours
- Number of antenna = 60
- Number of baseline = 1830
- Number of SPW = 1
- Number of channels = 2048
�[1m�[91m MS PROPERTIES: �[0m�[0m
- File Name: �[1m�[92m/myPATH/1688399183_cal_avg_data-3C273-corr.MS�[0m�[0m
- Column Name: �[1m�[92mCORRECTED_DATA�[0m�[0m
- Selection: �[1m�[92mFIELD_ID==0 && DATA_DESC_ID==0�[0m�[0m
- Pointing center: (ra, dec)=(12:29:06.700, +02.03.08.600)
- Frequency = 544.07 MHz [ 544.07 -> 1087.80 MHz]
- Wavelength = 0.38 meters
- Time bin = 10.0 seconds
- Total Integration time = 2.37 hours
- Number of antenna = 60
- Number of baseline = 1830
- Number of SPW = 1
- Number of channels = 2048
�[1m�[91m MS PROPERTIES: �[0m�[0m
- File Name: �[1m�[92m/myPATH/1688399183_cal_avg_data-3C273-corr.MS�[0m�[0m
- Column Name: �[1m�[92mDDF_PREDICT�[0m�[0m
- Selection: �[1m�[92mFIELD_ID==0 && DATA_DESC_ID==0�[0m�[0m
- Pointing center: (ra, dec)=(12:29:06.700, +02.03.08.600)
- Frequency = 544.07 MHz [ 544.07 -> 1087.80 MHz]
- Wavelength = 0.38 meters
- Time bin = 10.0 seconds
- Total Integration time = 2.37 hours
- Number of antenna = 60
- Number of baseline = 1830
- Number of SPW = 1
- Number of channels = 2048
�[94mSolving [Chunk 1/1] �[0m............. 0/11 [�[37m�(B�[m ] 00% - 0'00"