Summary of the SWAMP Log book

We left the morning to the T0 people since they needed to take a few runs more for final checks. In the afternoon we learned how to ramp up/down the HV, how to run the DAQ and Jean Paul tested that the recording and transfer of oscilloscope pictures can be done quite easily. The only problem so far is with Thomas' and Christin's ADC fitting program. We did not manage to compile it here. We will give it another try now after dinner.

The boxes we brought with us were delivered today to MAPO. Tomorrow we start the schedule of day 2 as planned.

The weather is sunny and -42 C (-63 C windchill corrected).

We are practically on schedule on day 2. The problem with the ADC fitting program has been solved. It is now run as a script within ROOT instead as an stand alone program. This avoids the linking problems.

We have taken a run for 12 hours that will serve as the reference run before any hardware is changed. Furthermore waveforms of strings 14 to 19 and also of 1 to 4 have been taken and saved. A homepage with waveforms will follow soon.

The only thing that was not achieved from the schedule was the measurement of the gain and delays in strings 14-19. The pulse generator burnt down just as we were going to start. Jean Paul and Mats spent some time trying to fix it and then trying to set up the system with another generator.

Jean Paul, Arvid and Mats have been measuring the delays and gain of strings 14-19. This is not finished yet but will be continued tomorrow. Christin, Thomas and Carlos spent the day measuring the spark rates on strings 1-4. We will do strings 14-19 when the other team finishes with the gain measurements. We have been using 3 channels of the oscilloscope to measure at the same time one channel and the two adjecent ones. Typical rates are about 2 per minute, with many quiet channels. The maximum we measured was 5. There is practically allways cross-talk in adjecent channels when a spark is present in the reference channel.

We still have to go through several channels, so we will make a table when we have all numbers and a web page with some screen shots we took. Probably this has been noted before by someone, but we observed sparks when someone just walks by the swamp crate.

We all have used the OM database to check that the cabling is indeed correct. For the moment no contradictions between the database and the actual cabling have been found.

From the experience gained today it seems likely that the work schedule for when the new crates are here is a bit tight. Measuring the spark rate is a VERY time consuming job, and for a full crate as the one housing strings 1-4 crate it takes a whole day. It will take less time for strings 14-19 since they have less channels connected to the SWAMPs.

Today we had a power outage in the station for about 15 minutes which disrupted work for about 45 minutes. One needs to power everything down properly before the power is cut and bring it up gently again afterwards. We ended up losing no SWAMPs but one of the flat monitors and a printer, which meant additional delays substituting them.

Today we finished measuring the gain on strings 14-19 and finished the spark rate measurements on strings 1-4. There are about 7 channels of string 13 in the same crate as strings 1-4. Waveforms of these were also measured.

A web page with summaries and plots of the measurements now exists. We will send out the address as soon as it is hooked up to the general amanda page here and available for external users. Right now it only works in the local network.

There was quite terrible weather today and getting to MAPO was tough. Strong wind (-62 C windchill corrected) and very low visibility.

Today Carlos, Thomas and Christin finished measuring the spark rates in strings 14-19, and the delay and gain measurements of strings 1-4 were completed by Jean-Paul, Arvid and Mats. The measurement of the gain of the delay channels will be finished tomorrow.

A web page is now available that contains the result of the work. It will be updated as work progresses. Christin and Thomas sent a mail with the URL

Please report any problems viewing it from the northen hemisphere since it took some fiddling here with directories until it worked locally.

We have come to the conclusion that what we thought were sparks in one channel and cross-talk induced in the adjacent channels is just electronic noise picked up by the crates and has nothing to do with the SWAMPS themselves. We conneced the three channels of the oscilloscpe to the propmt output of three different channels: two in the same rack (#9) but different crates, and the other in rack 2. We saw coincident signals with this setup which indicates general noise picked up by the electronics. Not counting these 'events' and selecting only the typical spark waveform shape, the actual spark rate of the swaps is negligible. We observed only 3 sparks in strings 14-19, and all in the same channel, with amplitudes about 50-100 mV. We did not observe cross-talk in these spark 'events'. The external noise pickup shows a higher rate, with amplitudes reaching up to 200 mV and quickly fluctuating. We will put screenshots of all these categories in the web page soon.

We have two casualties: Jean-Paul and Carlos are not feeling well. We will try to rest tomorrow. Todays weather is sunny, windy and -34 C (windchill -61 C).

We have taken today off. With Jean Paul being evacuated to Christchurch, Carlos with a strong flu and with no new crates here yet, we decided to take it easy. Besides, the T0 people asked us for some time to make some muon runs, and we thought today was a perfect occasion to give them control of the detector.

We will finish with the gain measurements on strings 1-4 tomorrow (only a few channels left) and we will invstigate further the general electronic noise pickup by the crates. For the rest, we are now waiting for the new crates.

We finished with the gain measurements of strings 1-4. The new crates are supposed to fly today from Christchurch to McMurdo. I told Darryn to ask for high priority for the delivery to Pole.

Since we had time, we started to take waveforms of strings 11-13. Probably we will have time to do the same measurements on these strings as we did on 14-19 and 1-4 before the crates arrive.

We are still waiting for the new crates. They have been given the highest priority, but still, it seems to take its time.

In the meantime we have taken waveforms and measured the gain, both for delayed and prompt channels, of strings 11-13.

Christin and Thomas have mirrored the SWAMP page to Stockholm so it can be accessed by the nothern hemisphere people without having to rely on the satellite connection. Address: http://www.physto.se/~amanda/swamp/swamp.html

The 4 boxes with the new crates have arrived this morning to the Pole and have been delivered to us in the afternoon.

In the morning Christin, Thomas and Arvid finished measuring the delays in strings 11-13. So, summarizing: up to now we have measured:
- gain of the prompt and delayed channels
- delays of the prompt and delayed channel
- spark rates (except strings 11-13)
- taken waveforms
on the crates of strings 1-4, 11-13 and 14-19.

In the afternoon we disconnected all cables from the SWAMPs of strings 14-19 and dismounted the crates. Mats labeled the delayed output cables with a red band to easily distinguish them from the prompt cables. We took our own note of cable labels that we will compare with the official database. Christins and Thomas have included this 'private' label database in the SWAMP web page.

We will let the boxes accomodate to MAPO temperature during the night and start tomorrow morning with unpacking and mounting the crates for strings 14-19. We will start also tomorrow with the shifts, day and night. We did not feel the need until now of doing so since we could carry out all of the measurements we did by working in paralell in two groups.

Christian Bohm has arrived today and Mats has started to work in the ICETOP project.

Weather report of today: -34.9 C, Windchill -49.8 C, Physio Altitude 10388. ft

We unpacked the boxes and mounted the first crate of strings 14-19 (this took its time...) We have put the delayed chips on the new channels. Each delay chip goes into the same slot as in the old configuration. We have set the gain in these channels to the old values. Due to a problem with the local network we could not set the HV and so spark rates will have to be measured by the next shift.

We have started two shifts as today: Christian, Christin and Thomas in the day shift and Carlos and Arvid in the night shift. Today we had Thanksgiving dinner with its delicious deep fried turkey and lots of NZ wine.

The two crates for strings 14-19 (crates 09 and 10) are now mounted on the rack. We have set the gain to the old values and we have been taking sparking rates over the night. We did not find any problem, except for one card which sparked once every two seconds or so. The rest of the cards were very quiet. Results will be posted in the web page soon. We will finish taking spark rates on the whole crate and then will exchange this card for a quiet one. Quads are not connected yet.

The offset found when working with crates 09-10 has been traced to the power plug used to plug the oscilloscope. If the scope is plugged to a different outlet, the offset disappears. Which means that the plug we used first has a different ground than the crate.

We might extend our stay here for a couple of days since we were waiting a few days for the crates. I talked to Alex this morning and there seems to be no problem on leaving the 7th here and taking the flight on the 8th to Christchurch.

Until now we have:

- Finished taking spark rates. We have not noticed any problem with this. No sparks or cross-talk.

- The problem with the sparky channel was traced to a loose screw inside the pickup box. The screw removed, the problem solved.

- We have connected the quads and taken waveforms. We have a small problem with a few channels: we have some 'too good' waveforms taken before the change, with the old setup. Look for example at waveform b14_04 in the web page.

- Waveforms for strings 14-19 before the change were taken by Jean Paul, so we do not know right now how he took this one (and a few similar ones). Some averaging option in the scope? Taken from the delayed channel? They really seem too good for a real pulse.

The new SWAMPS give better waveforms and nicer pulse shapes with better defined peaks. All noise levels are very similar, about 20 mV. Of course you cannot beat a waveform like b14_04. In this case the new SWAMP gives the normal waveform which taken at face value is not as good as the before 14_04, but we suspect b14_04 to have been taken with some non standard scope options.

We have come across some problems of a more serious basis in a few channels:

Channels that present more noise now than before. We have traced the problem to the pick-up boxes. Exchanging the pick-up card for the problematic channel has solved the problem. This is a quite time consuming task, involving uncabling in the backplane and changing the pickup card. Without Christian we could not have fixed these kind of problems since none of us was skilled enough to change a pickup cards! We have fixed 2 of these noisy channels in this way, which were unusable otherwise. There are still 2 or 3 more channels that are usable, better than before, but still noisier than the average (look at a14-29 in the web page, for example). One could think of changing the pickup card in these channels also and improve the noise level.

A second problem has been with OM 8 in string 17. The HV gives an error on this channel and does not bring the HV up in it, or if it brings it up, the current measured across the OM circuit is 0. This is a problem which is not completely reproduceable. We have turned on and off the HV several times to make the above mentioned repairs, and sometimes the HV has not been up in this channel, and other times it has been up to thenominal value but the current reported is 0. We will investigate this next.

In summary: The new SWAMPS give lower noise levels, better pulse shapes, and are more uniform across the channels. The few problems that we encountered have been fixed by changing the pickup boxes. You can take a look at the mirror page where the new waveforms should be now, called a(str#)_(OM#).gif, to be compared with the ones taken previously, b(str#)_(OM#).gif. (well, a few of the new ones have not been transferred yet, but there are enough on the page to get a good idea.)

Timescale: Here we come to the critical point. My view is that we should keep the new SWAMPS for str 14-19. They are better. We will follow them for stability in the next days, but I expect no surprises. What to do next? The main point is that it has taken us about 4 days to set up, measure and evaluate these two racks. This includes repairs, which involve uncabling-recabling and soldering and are time consumming. And we have been dealing with 58 connected channels here.

I think we will have time to exchange one more crate, or at least aim for this, once we have finished evaluating 14-19. Aiming at changing all 1-4 and 11-13 seems a bit too ambitious at this point (unless we extend our stay here for a week more . What about then overlap with the next batch of people comming for TWDR... etc?)

The day shift found the problem reported before of one channel not getting the HV. There was a problem in the quad connector. It was opened and fixed. There are no explicit problems now in the new SWAMP racks for strings 14-19. We have reconnected all the detector and we leave a normal muon run overnight

There is a small problem with one old swamp slot on crate 1, rack 1 (strings 11-13). The fuses blow very frequently. We will address this tomorrow.

So by tomorrow evening, after the muon run has been analized, we will have the final basis for a decision. We will send you a summary of the performance of strings 14-19 and update the webpage with the latest numbers.

We have also re-taken waveforms in strings 1-4, since this is not very time consuming and we were not sure how Jean Paul had done it. We will put these updated waveforms on the webpage also.

During a few hours last night we left a muon run going. It was analyzed during the day shift and at shift change we had a meeting to summarize the performance of the new SWAMPS in strings 14-19. We all agreed that they are an improvement w.r.t. the previous ones and decided to continue with the exchange of strings 1-4. Old and new waveforms of strings 14-19 are in the web page and ADC plots will be soon to compare (only the old ones are there now).

During the night shift we exchanged the crate 1-4. We took special care in documenting the cabling since now we have 20 channels per slot instead of 16. Delay chips from the old cards were put on the new cards of the string 13 slot that is in this crate, since these cards did not have delay chips.

Next: we will measure spark rates and waveforms on then new system and make another muon run including the new 1-4 crate.

We have mounted and cabled the crate with strings 1-4+13. We have measured sparks and taken waveforms. Everything seems ok except for 4 channels, that are dead now and weren't before. During the night we exchanged the SWAMP card on two of them to see if the problem was there. It was not. We also connected these channels to another SWAMP slot and then they were ok. So it has to do with the SWAMP system and not with the OM or the HV. We will try to check the pick-up cards tomorrow.

We did not encounter any problem with the adaptors to string 13. We first made a map of which pin was connected to which cable in the adaptor. Since the cables have different colours, we then had a systematic mapping for the rest of connectors. Then we put 100 V in one OM and measured the polarity and map it to the direction the connector should be plugged. So we plugged all the slot knowing what went were and in which direction. We left a muon run going through the few hours left until the next shift arrives.

We can say for sure that strings 14-19 and 1-4+13 are performing satisfactorily and there is no reason to go back to the old system. We might have even time to change 11-13. It depends a bit on how much time it takes to fix the remaining channels on 1-4. We have extended our stay here for a couple of days more than scheduled. We are now scheduled to leave on dec 7th.

We have continued today with changing strings 11-13. The day shift fixed the few channels on strings 1-4 that posed problems and that was just due to badly connected pickup cards. The newly taken waveforms for strings 1-4 and discharge measurements are in the web page for comparison.

The improvement is not as spectacular as for strings 14-19, but the different channels are more uniform now and we have recovered 3 channels that were dead before. Noise levels are comparable before and after the change. Slightly better now overall. Also the system seems quieter in terms of dicharges now.

During the night we have mounted the crate for 11-13. We will measure spark rates, set gain to old values and take waveforms during the next shifts.

The DAQ is extremely unstable these days and we have not been able to take a long muon run for evaluation of ADCs in 1-4. The one left running yesterday after the night shift crashed due to a power outage. We will try tomorrow during the day (when help from WOs is available) to make a muon run excluding 11-13, while we work on this crate. But at least we will have 1-4 and 14-19 in the run for evaluation.

If no serious problems arise we will have time to leave the system running with the new SWAMPs in 1-4, 11-13 and 14-19, being confident that the new setup works. We have a week now to make the measurements in 11-13 and follow the system.

Summary: As of today, Dec 3, we have exchanged all three crates, strings 14-19, 1-4 and 11-3 (we did it in this order, checking that the exchanged one worked before proceeding to the next). All are working fine. We encounter minor problems with few channels (order of 5) in each crate that have been fixed. They were mainly related with pick-up cards, and a minority with the connector itself or with the SWAMP card. In any case, exchanging the cards or opening and fiddling with the connector solve the problems. We can say that right now the system is working without problems with the new SWAMPs installed.

We will devote the remaining 4 days on training the winter overs in exchaging SWAMP cards, we will try to improve a few channels that are functioning satisfactorily but that might be further improved by changing cards or assuring the connector is solid. We might also try to see if previously dead channels can be brought to live now. These extra 4 days will allow us also to follow the stability if strings 11-13, that was the latest changed (yesterday), and the stability of the system overall. We also have to update the channel database since the crate/slot/cable mapping has changed.

We think it has been very succesful 3 weeks. Strings 14-19 and 11-13 are REALLY improved. Strings 1-4 are comparable in terms of noise but of course easier to service, and the waveforms more uniform across the channels.