November 5

November 6

In the morning we measured the noise levels for the two SWAMP slots we wanted to replace, and for one reference slot. The two slots to be replaced contain channels 5-1 through to 5-20 (slot B10 13/14) and 6-5 through to 6-24 (slot B10 11/14). The reference slot contain channels 5-21 through to 5-36 and 6-1 to 6-4 (slot B10 12/14). We made three measurements in total between nine in the morning and two in the afternoon.

In the afternoon we discovered that on two of the pick-up boxes a connector had been damaged, probably because the box had been dropped. When the boxes were opened we realized that two cards were loose in one of the boxes.

After replacing the two boxes we measured the noise again and saw a general increase of up to 10%, in some channels the increase was even much larger. However, the VLF signal is completely blocked.

The temperature stayed around -49C all day, and the wind was 12-15 knots.

November 7

November 8

Run stopped at 20:07. We once again measured the noise in the channels that got new pick-up boxes yesterday, and it seems stable. In channels 17 and 18 in B4 01/04 there seems to be a sine distortion that was present already before the change. Two HV cables behind B4 02/04 were badly marked (hard to read/label falling off) so we put new yellow sticky labels on them (33 and 36).

After stopping the run we shut down all HV and all SWAMP crates (front and back) and started replacing all pick-up boxes. The pick-up box to the far right in crate 9, rack 10 (behind the SWAMP module marked 'spare') has a faulty HV connector on channel 20 and hence channel 20 (which is empty now) should not be used. There's only one quad connected to this module, along with two HV cables, but in the front there are lemo cables connected to several channels that are empty in the back. In rack 5, B10 05/14 also has a faulty HV connector on channel 20 and hence channel 20 (which is empty now) should not be used. There are two spare pick-up boxes, but both have damaged HV connectors in some channels.

After replacing all pick-up boxes we ramped up the high voltage and started a new run 2004_313_8094 at 07:04:21. After a few seconds we got an HV error in 1458-2004, channel 11-1 (OM 656, HV cable red_19_18). We disconnected the cable and disabled the channel. Then we got a HV error in 1458-2002 in channel 4-11 (OM 367, HV cable Red_J09). We disabled that channel too. We'll deal with the problems tomorrow.

November 9

Dealt with the HV errors. HV cable Red_J09 was loose, after plugging it in properly the HV error disappeared. Found an error in the hardware database, channels 11-0 and 11-1 in 1458-2004 are swapped, but the HV values are correct. In reality it's connected like

After removing a cabel tie the HV error disappeared and the channel got the right voltage, but there was no current flowing. Changed pick-up card for the channel which made no difference. Swapped the two HV cables Red_19_19 and Red_19_17 at the power supply, then the problem moved with the cable (from chn 11-1 to chn 11-0), i.e. the HV channel from the power supply is working. Then put the cables back and instead swapped Red_19_19 and Red_19_17 at the pick-up box. Then suddenly both channels worked and produced waveforms. We leave it like this for the time being, but note that Red_19_19 and Red_19_17 are swapped at the pick-up box (Red_19_19 is now connected to SWAMP channel 9 and Red_19_17 to channel 11) and that the HWDB is incorrect.

Changed bandwidth from 10MHz to 7.5 MHz in channels 5-1 to 5-20 and measured noise again. There seems to be a general decrease in noise rate (0-30 mV) that is not seen in the reference channels 5-21 to 6-4. Hence the bandwidth was changed for all channels in string 5, and for channels 4-21 to 4-26 from 10 MHz to 7.5 MHz. All HV channels were enabled. The VLF veto was turned off and a new run 2004_314_8095 started at 04:26:22.

November 10

Run 2004_314_8095 was stopped around nine in the morning and a new run 2004_315_8096 was immediately started.

November 11

The current run was stopped at 21:40. Channels that seemed to have died after the upgrade:

Furthermore, channel 596 (dAOM) became very noisy, but is left for now. Ethan mentioned two OMs that are known to be dead that could be investigated, OM 67 (can't maintain HV), and OM 438 (has optical read out but the fibre is dead, can be switched to electrical but has no SWAMP card at the moment). After Steffen moved all channels on M1 to one of the 1458's, channel 414 has HV problems, it did however work fine in run 8095 so the problem is not directly connected to the upgrade.

All new pick-up boxes in B4 were replaced with the old ones, in order to get the noise rate back to normal. Strings 1-4 do not appear to pick up the VLF signal even with the old pick-up boxes, which can be seen in run 7926 when the VLF veto was accidentally inverted.

When ramping up the HV, the following HV channels had problems:

Since Steffen is working with the HV we couldn't start a new run.

November 12

Stopped run 8099 in the morning and started run 8100 right away. After some initial studies of pulses (about 20 minutes) the run was stopped. The two dead channels from yesterday (94 and 404) were investigated and fixed:

A new run 8101 was started before lunch. It was stopped again after lunch (01:49) in order to debug channels 67 and 414. We tried changing the HV cables for OMs 67 and 414. For OM 67 it had no effect (it still can't maintain HV), but so far OM 414 seems to be able to sustain HV with the new cable. OM 414 now has HV cable Red_G_30 (instead of Red_J_28).

Looking at monitoring for run 8099 shows that the noise rate for B4 has decreased again. However, the flare rate has increased up to ~3.5% in intervals. Also, the ADC info rate has increased for B4 after changing back to the old pick-ups, and is back to the same values as before the upgrade.

OM 48 has a worse ADC peak than before. The value has only decreased slightly, but there seems to be more noise.

OMs 655 and 656 have changed ADC peaks. For 655 the peak value has increased drastically, whereas it has decreased for 655 (basically no peak at all). The change occurred in run 8097. Maybe the HV cables have been swapped?

There are still bumps in the TDC spectra for some channels.

November 13

The day was dedicated to investigating the increased number of secondary TDC bumps, and the increased flare rate and it's possible VLF correlation. About 10% of the channels have new or larger secondary peaks in their TDC spectra. In a few cases strange peaks have instead disappeared the the spectra look better. The origin of these additional or increased peaks is the VLF filter that gives a larger overshoot.

After looking closer at several channels when the VLF was running, we conclude that no channel in strings 5-19 is seriously affected by the VLF and the largest influence observed was in the order a few mV (<5 mV). However, B4 does pick up the VLF signal as a 50 mV sinewave. We once again replaced one of the old pick-up boxes (channels 1-1 to 1-20) with one of the new, to see what happened. Sadly, the VLF signal is still present with the new pick-up box, but the amplitude is reduced to 20 mV peak-to-peak. We replaced the remaining B4 pick-ups with new ones. This will however increased the noise rate, but maybe we can solve that by reducing the high voltage. New high voltages will be obtained by comparing ADC spectra for runs 8090 (before pick-up change) and 8094 (new pick-ups in B4).

We've started a new run 8105 to see how the new pick-up boxes deal with the VLF (that will be turned off tomorrow).

November 14

Unintentionally, the SWAMP crate for B4 was turned off during run 8105. This morning the run was stopped, B4 turned back on, and a new run 8106 started.

November 15

The noise only decreased slightly and in some ADC spectra for B4 the VLF can be seen (mostly OMs 1,2,41,42). Now the VLF has been turned off and will not be turned back on again until January. Looking at the monitoring for run 8107 reveals that OM 307 has problems (supposed to be optical), and OMs 655 and 656 still have strange ADC spectra and the suspected HV swap must be investigated. The dead time was 16% and the flary percentage was back down to 0.23%. OM 307 has electrical read out, but no pulses.

Run 8108 (default HV) was stopped and all B4 HV were increased by 40 V and the B4 thresholds were set to 140 mV in another attempt to decrease the noise. Run 8109 started.

In the afternoon run 8109 was stopped for channel debugging. The pick-up card was replaced for OM 307. OM 307 is now working.

HV cable Red_Q_8 was connected to the wrong SWAMP channel (which means that OM 562 got no HV). It has been corrected.

Furthermore, it turned out that channels 11-1 and 11-2 in power supply 1458-2004 were swapped at the power supply. This explains the ADC peak shifts in OMs 655 and 656. Now we've connected it according to the HWDB again. We tried swapping Red_19_17 and Red_19_19 back to their HWDB positions, and immediately got a HV error. We swapped them again, and the HV error disappeared. It's now connected like

We got another HV error in 1458-2004 in channel 8-2 in the HV crate. We unplugged and replugged the cable and the HV error disappeared. We have started run 8110 which has the same settings as 8109.

Finally we've found some inconsistencies in the hardware database. Red_Q_7 is connected to OM 678 instead of OM 561. Red_19_40 is connected to OM 677 (dead) instead of OM 678. In practise this means that OM 678 gets 1330 V instead of 1410 V?!? This should be investigated and the HWDB updated.

November 16

Put back HV cable Red_Q_8. For channels 561,562,678,679 there are discrepencies between reality and HWDB that seems to have arisen during the winter (according to the log).

November 17

November 18

We briefly looked at the monitoring for our stable run 8128, and it looks ok.

New TWR testruns: new HV settings for strings 5-10,14-16, new thresholds 80mV for all electrical channels in 5-10,14-16. all other channels have default values. B4 same as for stable run. Exceptions: 88,91,99,114,115,116,118,119,128,130,233,246,289 have 140mV, and chns 159-174,267-282 have 100mV.

We've packed the things that are going back to Stockholm:

November 19

November 20

We started a new run 8166 with new HV settings for some channels in the range 430-552. After roughly an hour we stopped the run, and started a new run 8167 with updated HV for channels 106,114,115,246. We also changed the threshold for OM 446 from 50mV to 80mV because it has in fact electrical and not optical read out. Furthermore we changed the threshold for channels 463,465 from 300mV to 150mV. After starting the run we briefly checked the rate of OMs 463,465 and it looked ok. The trigger rate was around 88 events per second, we're still waiting for the monitoring file.

We looked at monitoring for run 8164 to see if there were any problematic OMs. OM 386 had ADC but no TDC (starting in run 8150). It turned out that the TDC cable had been disconnected. However, the lemo cabel connected to the prompt output for OM 386 is marked '42_B' in one end (the end connected to the SWAMP) and '41_B' in the other end. OM 515 had been dead since run 8161. We measured the noise, and it was less than 1mV peak-to-peak which lead us to believe that the SWAMP card was broken. We replaced the card by a spare and the channel was brought back to life. The gain will have changed though and must be checked.

Earlier we discovered that OMs 561,562 and 677,678 were not connected according to the database. According to the winter-over log, quads and HV cables were moved during winter (July) to check if there was a problem with one of the SWAMP channels. This is the mess we found: OM 561 had OM 562's HV cable. OM 562 had no HV cable. OM 561 was read out as OM 562. OM 678 had OM 562's HV cable. OM 678 was read out as OM 561. OM 677 had OM 678's HV cable. OMs 562 and 677 were not read out. In the table the black entries show what we found, the red entries how it should be according to the hardware database, and blue entries show how we reconnected everything.

OM 677 is a dAOM and shouldn't have a normal HV cable at all, but it did. First we reconnected everything as it was in the HWDB, but then there was no current flowing for OM 678 (HV cable Red_19_40). We then suspected that there might be a swap inside the quad so that OM 678 was on top of OM 677. Hence we put back HV cable Red_19_40, and then the HV channel was working again and we got pulses. This lead to the final set up marked with blue in the table. We confirmed that we actually read out the right channels, by putting HV on one channel at the time and looking at online histograms while doing a short test run.

To summarize, OM 561 was working but got the wrong HV (1650V instead of 1330V) and was read out as OM 562. OM 562 was dead because it had no HV. OM 677 had a normal HV cable eventhough it is a dAOM and was read out as OM 561. OM 678 got the right HV all along and is read out optically .

The string trigger was disconnected, probably accidently by someone's boot, Steffen reconnected it. When ramping up the HV we got an error in channel 8-2 in 1458-2004. We replaced the HV cabel Red_Q_18 wit an unused cable F_08 and the error disappeared.