Wednesday, September 25, 2019

SeaBird CTD tutorial series

Last June we (credits below) ran a SeaBird CTD workshop for the Israeli oeanographic community. While preparing for this workshop we prepared a series of tutorials in the form of blog posts with screencasts combined in them. Looking at the final product we believe those tutorials might save time for anyone beginnig to work with SeaBird products so we share them here.

  1. This post (link) will teach you the basics, how to connect, test and sample using SeaBird CTD.
  2. When you will try sampling you will probably have some problems, and this post (link) will help troublshoot and solve problems.
  3. Sometimes, you will have to reconfigure your CTD, this post (link) will show you how to use the software that does that (Sea-Term)
  4. When sampling and troublshooting you will have to configure your software, this post (link) will show you the different SeaBird file types and how to configure using each of these files.
  5. Once you left all of the operating and sampling behind, you will want to export the data to do oceanography, this post (link) will help in viewing your CTD data using ocean data view (ODV).

Tuesday, September 24, 2019

Post processing CTD data using ODV

In this post I will give a tutorial explaining how to prepare SeaBird CTD data for analysis using ocean data view (ODV).



Both ODV and SeaBird CTD are near Oceanographic standard, but getting used to processing the CTD data for ODV takes some time. With the hope of saving you some of this time I detail the procedure we usually take at the Israeli School of Marine Sciences.

Step one: 
Convert the hex file downloaded frome the CTD or saved while sampling using SeaSave software to .cnv file. This step is conducted using the Data Conversion option in SeaBird data processing software (SBEDataProcessing).

Step two:
Align the CTD data. Since the CTD instrument is a combination of several sensors, each having a typical time lag the readings in one data scan does not really represent a single point in time for all water quantities. To compensate for this, CTD alignment is performed using the Align CTD option at the SBEDataProcessing.


Step three:
Average the CTD data by depth. Many times, having data values at fixed depth intervals (eg. 1m) is more convenient. To achieve that we use the Bin Average option in SBEDataProcessing.

Step four:
Import CNV files to ODV and validate the CTD measurements.


Step six:
Many times you sample discrete samples on the same xast you sampled using the CTD and you want to plot the data together. To do that you will have to:

  1. Export the CTD data to spreadsheet
  2. Open the spreadsheet externally (in excel or other spreadsheet software)
  3. Add the discrete parameters and samples to the spreadsheet
  4. open the spreadsheet in ODV. 


Last step (seven):
Import the data to ODV and plot it. This is beyond the scope of this series.

Thanks to my teacher Gitai Yahel who taught me this process.

Tuesday, June 18, 2019

The Sea-Bird CTD configuration file (XMLCON), and other file types.

The instrument configuration file

Instrument configuration files are files located on your computer and determine how the CTD signal should be interpreted. The configuration file (xmlcon) enables the PC to correctly interpret the data file/stream. These files can be edited using a SeaSoft program (image 1) either by double clicking it on the windows explorer or by pressing "configure inputs" in SeaSave and some other SeaSoft programs.
Image 1: instrument configuration file interface on SeaSoft.
  1. In the top section of the configuration editor, the general setup of the device is determined. It is crucial to ensure that the sampling mode (Profile or Moored), the number of "External voltage channels", and the "Serial RS232C" settings match the instrument setup. Mismatches will results in a "Sacn length error".
  2. In the middle section of the configuration editor, the file specifies additional data that is added to the sampling file/stream. Things like GPS data (in NMEA format).
  3. In the bottom part of the file, the number of sensors that are mounted on the CTD, Their type and position is determined. This section of the configuration is the most important because it determines the calibration coefficients and therefore the accuracy of the sampling. By double clicking a sensor in the list you can open a specific sensor calibration file (image 2) where you can change an instrument calibration.
Image 2: Calibration window for specific instrument.
    The .xmlcon should be changed whenever you change instrument configuration on your device or whenever you calibrate a sensor. Changing Simple calibration can be performed by double clicking an instrument, Then you can change the calibration coefficients either manually or by by importing an instrument calibration file (it's a specific XML file). Changing the order, number or type of mounted sensors (white lines in image 1) can be performed by right clicking a line and then "select new voltage sensor".

    When sampling with a CTD using SeaSave, Some of the configuration file is added to the instrument output (.hex file) as a header. you can review this section by opening a hex file using a text editor.

    The .psa file

    .psa file (short for program setup files) are file that configure a singe seabird program. each of the seabird software suite programs (SeaSoft) has its own .psa file. The functioning of the software is defined using the .psa file. For instance, in the image and screencast below you can see the serial port communication definitions in SeaSave, things like number of serial ports used by SeaSave, their port numbers and baud rates. all of those definitions are save in the .psa file.

    Image 1: Some of the content defined using the SeaSave .psa file
    When exiting one of the SeaSoft programs and if you made any change to the setup file you are asked whether to save the changes. For instance:

    1. When sampling using SeaSave you change the output data file name every time and therefore either you save the setup file or no it doesn't matter because you will have to change it again 
    2. You used your friends CTD once and changed the baud rate, its advised not to save the .psa after sampling. 
    When asked to save the .psa file. think before you push "YES" (or "NO")

     The dsa file

    The dsa (device setup file) is a file that defines the configuration of items within the SEaSave data window. Which sensors to display in a window, which variables etcetera. the SEaSave .psa file is linked with each windows .dsa file.


    Sunday, June 16, 2019

    The seabird CTD terminal (SeatermV2)

    SeaTerm is one of the SeaSoft components that is used to communicate with the CTD devices. The main activities we perform using SeaTerm are:
    Inquiring the status and configuration of the CTD
    Programing the CTD
    Uploading data stored on the CTD to the PC
    Once you establish communication with the device (screencast), you can send commands and receive output using the terminal. Seanding a command is performed either by typing it at the terminal window and then pressing enter or by double clicking the command name at the left panel and then pressing enter.
    When the magnetic switch is "On", SeaTerm only echos the device output. To use SeaTerm, Switch the device to "Off"

    Main commands inquiring the status of the CTD

    DS - this command will return the configuration status of the device (Which voltages are open), communication settings (baud rate), time and date, logging (internal memory) status, and battery voltage. this command is more comfortable than other status inquiries because the output is in simple text format and not in XML format. 

    Main device programming commands

    When using programming commands the interface will prompt for command parameters.

    DateTime - Set device date and time (For oceanographic applications, UTC time is preferable).

    SetBaud - changes the CTD-PC data transfer rate and therefore useful when uploading data (see below)

    InitLogging - Erases the device internal memory

    OutputFormat= This command will change the default sampling output format (when magnetic switch is "On") from hex to one of several formats. This is sometimes useful when troubleshooting your device or for some specific applications. BUT if the device is set to none hex output, you will have no communication with SeaSave.

    Profiling Mode commands - here you will be able to program the parameters of sampling in profiling mode, this is pretty straightforward.

    Minimum conductivity frequency (in the profiling mode menu) - many times we switch the device to "On" and we only want the pump to start working when the device reaches the water (to save battery power). We define a minimum conductivity frequency so that the pump will only start pumping when reaching salt water. This parameter should be set when sampling in fresh water.

    Testing commands - Using this menu you can view the the output of one sensor for testing purposes.

    PumpOn (testing menu) - Switches the pump to "On". This is useful for troubleshooting pump problems. To switch off use PumpOff

    Data upload


    Troubleshooting communication with the Sea-Bird CTD

    Communication problems

    Usually, if you just started using a CTD, your device will not let you enjoy an easy ride, faults will happen and communication problems are the most common.

    The easiest way to fix communication problems requires that you use a program called SeaTerm (we actually prefer using SeaTermV2). SeaTerm is a terminal software that communicates with the CTD device and is designated for programming the CTD (see the list of important SeaTerm features in another post).

    Before you start SeaTerm, you should know that serial ports, as opposed to USB can not be connected to two programs simultaneously, so be sure to close all other SeaSoft software before you start SeaTerm.

    When you open SeaTermV2, you will be prompted for the type of device. Once you selected a device (here we assume you are using SBE19PlusV2) you will be getting the list of Command for your CTD (left) and the terminal itself (right). On the top bar, press communication>configure. Here two types of communication problems can be figured out:

    Port: Check the device manager again to make sure that the port you use for communication is the correct one, and then switch the CTD to "on" using the magnetic switch, if you don't see a stream of characters on the terminal screen, you have some physical problem with your cable or the serial to USB converter.

    If you see a stream of characters that are not letters, you probably have the wrong baud rate.

    Baud: The Baud is the symbol rate, or more to the point the CTD to PC communication speed. if you press the drop-down menu, you can see that your device can operate in many Baud rates. If your CTD's Baud is different than the Baud the PC expects, the symbol stream will look very incoherent, once the correct Baud rate is used, the character stream will only include hex characters (0-9 and A-F).

    Once you have established communication it's advised to switch off the CTD disconnect and reconnect and then check the CTD by sending it the get system status command (just type GetSD on the terminal window and read the results). Once you are done with the terminal, make sure to close the program.

    Scan length error 

    Scan length errors occurs when there is an incompatibility between The CTD device definitions (conducted using SeaTerm) and the instrument configuration file which is stored on the PC and defines how to translate the device output.
    To save memory and battery power, the user can define the number and types of sensors used by the CTD. A small number of devices require less power and less memory (and also means  narrower communication band to the PC). If the PC configuration file defines different number (or types) of sensors than the number physically defined on the CTD the communication bands are inconsistent and a scan length error is received.
    To fix the error you will have to understand how sensors are numbered on the device and in the configuration file.
    On the CTD device, Sensors have no real meaning, all we have are voltages (volt1-volt6), each voltage is a communication port on the CTD you can open (using the volt 0...5=Y in SeaTerm which makes the number of characters in the CTD output longer) or close (using the volt0...5=N). Only after opening a voltage on the CTD, you can mount a sensor on that port and the data will be saved internally or transmitted to the .hex file on the PC.
    Image 1: location and number of ports.
    On the PC, when translating CTD output (electric current readings (V)) to sensor readings (such as temperature) and the number of sensors is defined on the top of the configuration file under "External voltage channels".
    The CTD output is cut into pieces, each piece is related to a specific sensor and translated to measured quantities (eg. Oxygen concentration) using the sensor definitions on the configuration file. To properly cut the CTD output, you have to correctly define the position the sensor on the CTD. This is done by correctly positioning the sensor on the list of sensors in the configuration file (image 2).
    Image 2: instrument configuration file (.xmlcon)


    AND here is the catch.  Although the sensors are related as voltages the numbering is different. Here a sensor number is numbered by the voltags position when only counting voltages which are open on the CTD.
    Example:
    We have a CTD device where we open voltages 1,3,5 and an Oxygen sensor is mounted in the top port (image 1). The oxygen sensor should be defined in the .con file under the volt 3 line.  


    Low battery

    If your CTD runs out of battery power while sampling it will switch off to prevent for inaccurate measurements. As a user you will experience communication because the PC will lose communication while sampling.
    To verify this, you can check the battery status using the DS command on Seaterm. If you batteries (Vmain on the command output) voltage is less than 9.7, This is probably the case and you should change batteries before you continue sampling. SeaBird produced a clip describing how to change batteries. Also, when communication fails due to CTD shutting down, you will see a comment "power off" at the end of the hex file produced while sampling.

    Saturday, June 15, 2019

    Water column sampling using Sea-Bird CTD

    Sea-bird product line is vast and divers, in this post we will only teach how to use Sea-Bird SBE19plus V2 in profile mode without a deck unit (Ship communication system), although many parts of this post will also apply to other instruments and to sampling while using a deck unit.
    This post was written in order to help people when sampling alone and is very detailed to achieve its purpose.
    External links are highlighted in yellow text.
    Establish communication
    Due to backward compatibility issues, Sea-Bird devices use RS-232 communication protocol. Such devices are using Serial port (Image 1, A) for communication with the PC. Although serial ports were the norm in personal computers for years, many modern computers are not equipped with such ports, to overcome this problem, we now use serial to USB converters (Image 1, B).  Since communication problems are common when sampling with a CTD, we highly recommend that if you are purchasing a computer to be used for CTD sampling, you will acquire one that has a serial port.

    Image 1: A - Serial port. B - Serial to USB converter.



    The converters comes in many sizes and shapes, but they all require installing a driver software. Usually the driver can be downloaded from the manufacturer website, or you can use a CD supplied with the device.

    After installing, plug in the serial to USB converter in one of your USB sockets, a virtual serial port is created. First, make sure that this port was created and find out the port number using "device manager":

    To open the device manger software, open your file windows file manger manager and right click on "This computer">properties>device manager. After opening the device manager, go to "Ports", expand the list of ports and identify you serial port (remember the number in brackets after the name). Since the naming varies with adapters and windows versions, you can use the screencast or have a look at image 2 to identify the converter's port number.
    Image 2: Windows device manager



    Software

    Seabird has a suite of software called SeaSoft which you can download from the company website. This suit includes the three programs that are usually used when sampling with a CTD:

    SeaSave - This software is used to acquire CTD data to disk and display it in real time. It can also display archived data. When using a deck unit, this software controls the operation of the rosette. Its important to mention that the way SeaSoft programs look and behave is extremely configurable and is controlled using .psa (program setup files) and .dsa (display setup file) files.
    SeaTerm - This program is used for programming the CTD and for downloading data saved on the device.
    SBEdataconversion - This program is used when post processing the data acquired when sampling. In a separate post we describe the data conversion process we usually perform.

    Before sampling, its essential to know three types of files:
    1. psa - program settings file
    2. dsa - display settings file
    3. xmlcon - instrument configuration


    For further info regarding those file read the designated post here.
    Before sampling with the CTD, preferably before you exit the office, make sure everything works:
    • Connect the communication cable to the PC (using the adapter if needed).
    • Check the connection by performing a dummy sampling:
      • Switch on the CTD using the magnetic switch.
      • Go to Real time data>start and then push the button to select output data file name (select a file that you can easily identify as test run)
      • make sure you use the correct instrument configuration file (its a good idea to use a date in the file name)
      • Push the start button and see that communication is established (otherwiseuse the troubleshooting post here).
      • Make sure that all the sensor are responding and produce sensible readings.
        • Blow into the pump to check the salinity, Temperature and Oxygen sensors that are connected to the pump.
        • Use your hand to cover optical sensors and see that they respond.
    • Exit SeaSave and switch off the CTD using the magnetic switch.
    When arriving to a sampling station perform the actual sampling:
    • Connect the communication cable to the PC (using the adapter if needed).
    • Check the connection before you put your device in the water:
      • Switch on the CTD using the magnetic switch.


      • Go to Real time data>start and then push the button to select output data file name. Its important to define a file name that is meaningful, the following lines I assume using the MichSave script we use for naming CTD files.


        • Run the MichSave script by double clicking it
        • Select the project number from the list
        • Write the sampling station name
        • SeaSave will be launched automatically, push the "select output data file name" and paste (CTRL+V). This will paste the automatically generated file name.
        • Select the "Begin archiving data when ‘Start Archiving’ command is sent" option
        • Make sure you are using the correct instrument configuration file (its a good idea to use a date in the file name).
        • Click start
        • Wait for communication
    • Lower the instrument into the water (preferably to a. 5m depth).
    • Wait for the instrument readings to stabilize.
    • Lift the instrument back to the surface
    • Click real time data>start archiving
    • Sample your profile
    • Make sure you are not lowering your device faster than 1m s$^{-1}$
    • Closely observe your readings while sampling!
    • When done sampling close the sampling by clicking real time data>stop

    Wednesday, June 12, 2019

    Naming script for CTD files - MichSave

    The missive is a bat script that helps document your CTD samples (to download it use this link). .bat files, are simple text files with a series of instructions of the type you type in windows command line, to make a text file into a script all you need to do is change the file name extension from .txt to. bat. For basic operation, you can watch this screencast:

    How to use MichSave

    In this post I will explain how the script is used and how to change the script code to suit your need.
    To run the script just place it on your disk and double click (you might get a warning message, we all know you trust me).
    After clicking a black screen appears:
    The MichSave script
    CTD sampling are usually not stand alone operations, but they are a part of a scientific project. This is why the script asks you to select a project code from a list. Is your project is not in that list, you can enter a name manually or edit the script (as explained later in this post).
    As each project usually includes more than one sampling station, after you enter the project code, the script will ask you for a station name. Type the name of the sampling station and ENTER.
    Sea-save will be opened automatically (if not, look into how to change the Sea-Save location later in this post).
    Once SeaSave is openned, from the top menu, press "real time data">start>"select output data file name" and then use paste (CTRL+V) and enter. The automatically generated name will now be your output data file name.


    How to adjust the script to what I need?
    In this section I will go over the script line by line and explain how to adjust the code for your needs. Microsoft notepad  is a software that every windows computer has and is suitable for .bat file editing. Since double clicking will run the script you will have to right click it and choose edit form the drop down list.

    Adjusting the date format (lines 10 and 11):


    set datetimef= %date:~10,4%%date:~4,2%%date:~7,2%-%time:~0,2%%time:~3,2%%time:~6,2%%time:~0,2%
    echo %datetimef%

    This line builds a string from the current date and time that will later be a part of the file name. They are depended on the way your system date is set up so you might have to tweak them a bit. This is best done by operating a command prompt (push the WIN key+R>type cmd>enter) then typing each of the parts that creates the script to see what is actually does.
    Short screencast here showing how to edit the date

    Adjusting the list of projects
    
    
    line 14 prompts the user to type a number to select a project.
    
    
    
    
    set param=%PROJCODE% 
    
    
    echo Select project \1-RIME\2-RECO\3-CO Chemical ocean\ 4-OC ocean cruise\ 5-Shira dispersal Or type free text for Project codes line contains the description text of the list of projects
    
    

    While line 15:

    set /p PROJCODE=?
    
    
    
    
    Indicates that once ENTER is pressed, the text the user typed will be stored in a 
    variable named PROJCODE.
    Next, lines 18 to 48 selects a project from the list of projects based on the number 
    the user typed. If no number is typed, the name entered manually is stored in PROJCODE 
    and becomes part of the file name.

    set param=%PROJCODE%
    
    echo %param%| findstr /r "^[1-9][0-9]*$">nul
    
    if errorlevel 0 (
     if %PROJCODE%==1 goto :1 
     if %PROJCODE%==2 goto :2 
     if %PROJCODE%==3 goto :3 
     if %PROJCODE%==4 goto :4 
     if %PROJCODE%==5 goto :5 
    )
    set PROJ=%PROJCODE%
    goto :selected
    
    :1
    set PROJ=RIME
    goto :selected
    :2
    set PROJ=RECO
    goto :selected
    :3
    set PROJ=CO
    goto :selected
    :4
    set PROJ=OC
    goto :selected
    :5
    set PROJ=SHIRA
    goto :selected
    set /p PROJCODE=now just paste the station name in seasave
    :selected
    So, if you want to change this section, all you have to do is change the text in line 
    15 and the corresponding item on the list:]
    line 15:


    
    
    echo Select project \1-RIME\2-RECO\3-Some New Project.......
    
    
    And the appropriate menu item:
    
    
    :3
    set PROJ=SomeNewProj
    goto :selected
    
    
    The last lines:
    
    
    
    
    rem ### Run seasave
    "C:\Program Files (x86)\Sea-Bird\SeasaveV7\Seasave.exe"
    
    
    Are running SeaSave, Sometimes, the location of the seasave.exe file is different. In
    this case you will have to edit the path to Seasave in the bat file.