ASP/GASP real-time observing

ASP/GASP real-time observing is usually very straightforward. These instructions will assume that you have set up the IF/LO system (using cima at Arecibo and Astrid or config_tool at Green Bank) to provide signal to ASP/GASP.

Logging In
Signal Levels
Configuring
Taking Data
Troubleshooting

Logging In

In about 4 or 5 windows, log into gasp or asp:
ssh -l gasp gasp0.gb.nrao.edu from anywhere, or
ssh -l asp aspmaster from remote.naic.edu
Password obtainable from any of us by phone.

Signal Levels

Make sure you are looking at the correct IF. On asp: "setif 2" (if you see no signal, get the operator to reset the abpp). On gasp: "setif gbt-b".
Adjust levels on asp/gasp. This has to be done in cima (AO) or cleo (GB). In cima, change levels in the "advanced if/lo power control" widget. In cleo (GB), once the IF rack is balanced, look at convertor modules 11 and 15, and adjust the attenuators if need be. More gain is also available on gasp itself (not asp yet) by logging into cpci as root and running set_gain, e.g.: "set_gain 0 10" amplifies the signal in polarization 0 by 10 dB.

How do you know when you've got the right power levels? Run "rms" on asp/gasp. (Gasp will ask you for a password for logging into cpci; this is available from any of us by phone.) "rms" will give an output that looks something like this but with larger numbers in columns 2 and 4:

Ref FWHM = 64.0
Ch FWHM0 dB0 FWHM1 db1
-------------------------------------------------
0 8.03 -18.0 2.88 -26.9
1 2.30 -28.9 2.31 -28.8
2 2.35 -28.7 2.06 -29.9
3 2.39 -28.6 2.39 -28.6
4 2.50 -28.2 1.99 -30.1
5 2.43 -28.4 2.42 -28.4
6 2.74 -27.4 2.58 -27.9
7 2.38 -28.6 2.32 -28.8
8 1.32 -33.7 2.35 -28.7
9 2.27 -29.0 2.28 -28.9
10 1.97 -30.2 2.07 -29.8
11 2.41 -28.5 2.31 -28.8
12 2.34 -28.8 2.32 -28.8
13 2.54 -28.0 2.45 -28.3
14 1.99 -30.1 2.45 -28.4
15 2.15 -29.5 2.08 -29.8
16 88.86 +2.9 95.41 +3.5
17 2.17 -29.4 2.12 -29.6
18 2.80 -27.2 2.86 -27.0
19 2.47 -28.3 2.50 -28.2
20 2.44 -28.4 2.68 -27.6
21 2.35 -28.7 2.28 -29.0
22 2.33 -28.8 2.46 -28.3
23 2.33 -28.8 2.27 -29.0
24 2.27 -29.0 1.39 -33.3
25 2.34 -28.7 2.36 -28.7
26 2.39 -28.6 2.41 -28.5
27 2.29 -28.9 2.44 -28.4
28 1.91 -30.5 2.15 -29.5
29 2.26 -29.0 2.48 -28.2
30 2.32 -28.8 2.03 -30.0
31 2.37 -28.6 2.29 -28.9

The 2nd and 4th columns are the fwhm quantization outputs for the two polarizations. For small-bandwidth observations (eg AO 430 MHz), only 2-3 channels on either side of channel 16 are expected to have much signal. For wider-bandwidth observations (eg GBT L-band), all channels should have good signal levels. We would like to aim to have fwhms of about 60 counts in each channel (the 3rd and 5th columns tell you the suggested change in attenuator dBs to get to that value). This is easily achieved for GBT observations thanks to the set_gain possibility. For Arecibo L-band observations done in parallel with the wapp(s), the signal levels will generally always be low, about 25 or 30. Unfortunate, but true.

No signal at Arecibo? (That is, the rms output looks like the above example?) Try getting the operator to reset the ABPP, which does the signal selection. Ask the operator to do "ctrl-c, off, on, slave" and try rms again -- this is a standard procedure for the AO operators, so there should be no confusion.

Configuring

There are two essential files for an ASP/GASP observation. From now on, we will use gasp0:/home/gasp and aspmaster:/home/asp interchangeably.

The first file is the config file, which needs to exist in /home/gasp/config/config.xml . (To be safe, also copy it to /home/gasp/arts/config.xml .) In practice, we keep loads of config files in the /home/gasp/config/ area, under the appropriate project codes. Select the config file you want to use.

For a cal scan, edit the config file, changing the "cal_scan" line flag from 0 to 1. Then copy the config file to /home/gasp/config/config.xml and /home/gasp/arts/config.xml .

For a data scan, edit the config file, making sure that the "cal_scan" line flag reads 0. Also make sure that you have enough dumps (n_dump) such that n_dump times t_dump is at least as long as your intended observation (in seconds). Then copy the config file to /home/gasp/config/config.xml and /home/gasp/arts/config.xml .

For a data scan, you also need the second essential file: a valid polyco file (for the whole observation) in /home/gasp/runtime/polyco.dat . Here are Rob's instructions for how to generate the needed multiple-channel polyco file:

Usage: makepoly psr centfreq bw MJD telescope tobsh
psr --- pulsar name
centfreq --- centre frequency of observation
bw --- total bandwidth
MJD --- current MJD (with fractional part) This is an optional argument -- if not given, it will assume NOW.
telescope --- telescope used (1=GBT b=85-3 3=Arecibo)
tobsh --- length of observation in *HOURS*
where centfreq means the centre of the entire bandwidth used.

It's important to get the centre frequency right. One way to figure it out is that the cima/cleo sky frequency must be 34 MHz (yes, 34 MHz, not 32 MHz) offset from the "rf" in the config file, in the direction opposite to that given by band_dir. The polyco centre frequency is then 32 MHz from "rf" in the direction given by band_dir. So for the case of 1802-2124, where the config file is /home/gasp/config/GBT06B-044/config.xml.1802.cgsr.24chans, (cleo frequency 1400, rf = 1434, band_dir = -1), we would type:
makepoly 1802-2124 1402 128 1 8
and this would generate GBT polycos for 128 MHz (32 x 4 MHz) centred on sky frequency 1400 MHz, valid for 8 hours from the time we hit "enter".

Important step needed for GASP (not ASP): type cgsr_send_all to send the config and polyco files to the cgsr nodes that we use for wider-bandwidth observations. Whether or not the cgsrs are used for a particular observation can be determined from the config file.

Taking Data

With the config and polyco files in place, and assuming the levels are good and (for a cal scan) the cal is pulsing, it is time to take data. Assuming you have 3 ASP/GASP windows available, one useful pattern is to do the following:

Windows 1 and 3: cd /home/gasp/data/GBT06B-044 (or other project code)

Window 1: start_asp

Window 2: stat_msg

Check stat_msg output for two or three (equal to number of data servers) identical start times. If the start times never appear, or do appear but are not identical, see Troubleshooting.
Window 3: once start times appear, you can run stat_asp to get a pgplot display of the accumulating profile for that dump. This gets flushed every dump, and with a certain delay, so don't be alarmed if it sometimes looks ratty. If gasp is working properly, the pulse peak should be pretty much lined up down all the channels, including the cumulative display on the bottom.
If you need to kill the observation before the last dump, ctrl-c in Window 1.
As Window 1 finishes and starting cleaning up, ctrl-c in Window 2 (stat_msg).
Also ctrl-c in Window 3 (stat_asp) and important kill both the pgplot window and the pgplot server.
Any window: stop_asp. There is also kill_asp which sometimes works better. In particular, GASP observations should be ended with kill_asp to make sure the cgsr machines stop GASPing.

Troubleshooting

Here are a few gotchas and suggestions for how to deal with them.

Start_asp and other scripts demand passwords when trying to log into the nodes. Probably the system has been rebooted and the master came up after the nodes. Cure: log into each node as root and reboot. That should fix the problem.

Cal scan: stat_asp shows a flat line. Make sure you have the pulsed cal turned on (cleo receiver screen, or wapp widget).

Cal scans and data scans all show garbage (eg sinusoids during cal), and you're sure you have the IF/LO set up correctly. Try power-cycling both the cpci computer and the analog downconvertor box, making sure both are in a powered-down state at the same time. In Arecibo, cpci is found in the blue second "new" asp rack, right above the 4 dataserver computers, and the analog box is directly above it. In GB, these are the metallic box with 3 LEDs on the front in the non-node gasp rack, and the small black computer above it. Both have power switches at the back.

Bad power levels (for example in one polarization): check the IF/LO configuration, of course. If bad levels persist, the option of last resort is to power-cycle the analog box and the cpci computer.

Start_asp window gives a seg-fault. Most likely there is an xml syntax error in your config file, so check there.

Start_asp window complains that polycos are not found. Things to try:

Make sure the pulsar name in the config file and polyco file match.
Make sure you have the correct centre frequency for your polyco file (1400 vs. 1402 MHz is a common error).
Make sure you have generated polycos for a sufficiently wide bandwidth. 64 MHz is fine for Arecibo, but for GBT L-band or S-band, you'll probably need 128 MHz.
If you were trying to take a cal scan, edit the config file to reflect this! Cal scans do not require polycos and should never complain about them.

Start times never appear in the stat_msg window. First, kill the scan and do stop_asp. A few things to try:
- Make sure you have the correct dataserver/channel-offset combinations in your config file; see here for current values. Including a dead ds in your config file will likely result in obvious complaints about "no route to host".
- Delete uncommunicative slave nodes from the config file, also zapping channels if need be.
- Check start_asp window for a polyco problem (as above).
- (For cal scans especially) try typing "go" in another window. This will probably start the dataflow, but may not produce good recorded profiles.
Start times appear but are different. Kill the scan, stop_asp, and try again.
shmem_net overflows in stat_msg window. Things to try:
- Check to make sure these error messages keep coming. A few (typically a couple of groups of eight) shmem_net overflow messages are nothing to worry about, as long as they stop after a few groups of eight. These may appear from time to time, in some dumps and not in others.
- If the overflow messages are persistent, kill the scan. You'll have to try again.
- One thing to check in the config file is that the number of channels being processed in a data server divides evenly into the number of slaves assigned to that data server (eg 4 channels, 8 slaves is fine; 4 channels, 9 slaves is usually not fine.) Add or delete slaves or channels as needed. High DMs typically require more than one slave per channel.
- GASP only: if you are using the cgsrs, it may be that Haydon's processing is running and interfering with GASP data acquisition. Log into cgsr0 as cgsr (contact Ingrid, Michael, or Bryan Jacoby for how to do this) and run the $PSRHOME/cvshome/hknight/csh_script/uberkill.pl script.
Pulse lines up differently in top and bottom sections of stat_asp display. Make sure you have the correct dataserver/channel-offset combinations in your config file; see here for current values.
GASP only: The bottom 8 white lines on stat_asp look funny and kind of like a square-wave. This often happens when the first 2 data servers use 1 gasp node/channel and the 3rd data server uses 2 cgsr nodes/channel. This seems to be a problem with stat_asp and not with the data collection, so don't worry (much) about it.

Ingrid Stairs

Last modified: Tue Aug 1 14:56:33 PDT 2006