Reading 400Msamples of Spectrum Analyser IQ Capture Memory in Python

Saturday, March 3rd, 2012

Not as the title suggests, but I’ve always been one to feel that to learn a new language you need a task or a project that can make use of it. I’ve been a bit of a stalwart of PHP since version 3, using it for solving those common tasks everyone needs to do on occasion such as parsing out a text file, comparing data, regular expression searches etc. PHP has even been used for controlling test and measurement, running test scripts and debugging customer problems in remote control which seems a little strange for a language written for the web.

Recently also found out that Phoronix the Linux benchmarking / test suite is written in PHP. It can be extremely useful as a general purpose language to achieve things quickly rather than needing to use a compiled language. I don’t want this post to be a PHP vs Python post, I’ve made good use of PHP over the years and still have projects ongoing that use it. Its just a view from the other side.

A number of customers make use of other languages such as Python to control test systems and automate systems and a colleague not too long ago told me of his efforts with Python to parse some web pages. So, it seems there was some mileage for me to take a look at it.

In this instance, the problem task was reading 400Msamples of IQ capture memory from our new latest and greatest analyser the R&S FSW. When performing a binary read of the data from the capture buffer in FLOAT32 format this results in 3.2GB of data to be read and stored. With smaller amounts of IQ capture, you could load the data into PC memory in one hit then write to disk. Although with 32bit systems, this is pushing it if you are running any applications at all – your likely to run out of RAM given the limit of around 4GB. So a quick solution was needed to buffer data to disk as it is read from the socket. Already having some examples of reading IQ data using PHP and C# with a socket connection to the analyser, this wasn’t considered too difficult to do in converting a script to Python but adding the file write in the process.

The implementation used for this case has a couple of modules. One module for setting up and handling the SCPI communication to the instrument over a socket called scpicontrol.py and another module called iqreader.py which handles getting the IQ data from the socket as either ASCII or Binary and formatting it / dealing with it appropriately to store to disk.

After the exercise, it has to be said Python is easy to learn and fast to develop in. Of course, if you want extremely fast socket communications the use of a statically compiled language win here for pure speed. Although the other part of the problem we did not know what OS the customer would use and we needed a solution quickly – in less than a day.

Python seemed to fit the bill for the task as its open source and cross platform and in this case it is also possible to run direct on the analyser with the interpreter installed (caveat: possible, although not necessarily recommended).

There are a number of other things about Python which, unless doing some specific Web work, will be my language of choice now for fast development and customer examples. After thinking about it, I’m surprised I didn’t pick up Python sooner.

  • Namespaces are implemented by default – so you don’t end up with long unwieldy function names like in PHP or accidentally overwriting a function that already exists.
  • Returns from functions in the Python Standard Library seem to be fairly consistent. In PHP this is far from the case, where a function may return for example true, false or zero. Of course false or zero are not the same thing and are interpreted differently, so you need to take care in some functions to check for !== false (not identical) rather than != false (not equal to) or you may get the wrong result. This of course is just one example, there are others I’m sure.
  • Object Oriented and procedural programming can be easily mixed, not necessarily a good thing, but makes for fast implementation. In PHP version 5 gets a new object model and namespaces – isn’t it a bit late at version 5 to be implementing these things?
  • Python is fast. After converting some of the PHP scripts I use for text file parsing, I’ve found Python seems to handle the same data in about half of the time. Of course, that could just be how the code is put together, but if Python assists in writing more efficient code, thats a good thing?
  • Python is self documenting which makes life somewhat easier :-)

fsxiqcapture.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132

# Main function
def main():
    # Get the Command Line parameters in first
    argv = sys.argv
 
    # Debug Args for testing various modes (overrides command line inputs)
    # Uncomment to use
    #argv = ["",""]
    #argv = ["-i","192.168.1.2", "-n","100","-b","ASCII","-f","iqdata.txt"]
    #argv = ["-i","192.168.1.2", "-n","1000000","-b","BIN","-f","iqdata.iqw","-a","1MHz"]
    #argv = ["-i","192.168.1.2", "-n","1000","-b","BIN","-f","iqdata.iqw","-a","1MHz"]
 
    # Check some arguments have been entered
    if len(argv) > 1 :
        if argv[1] == "-h":
            print "Program Usage:\n"
 
            #80 charachter line
            #-------------------------------------------------------------------------------
            print "This program captures IQ Data as IQ Pairs from R&S Spectrum Analysers"
            print "The data can then be easily manipulated and changed to a Waveform file"
            print "to run on an R&S signal generator by using Arb Toolbox (1GP62)\n"
 
            print "Command Examples:"
            print "fsxiqcapture.exe -h => Gives these instructions"
            print "fsxiqcapture.exe param1 param2 ...\n"
 
            print "-i   IP address e.g. -i 192.168.1.2"
            print "-f   Output File e.g. -f iqcapture.txt, default = iqcapture.txt"
            print "-b   Read Type e.g. -b BIN or -b ASCII, default = ASCII"
            print "-r   Resolution BW e.g. -r 10MHz, default = 10MHz"
            print "-t   Trigger setting e.g. -t IMM or -t EXT, default = IMM"
            print "-a   Sample Rate e.g. -a 10MHz, default = 10MHz"
            print "-n   No. Captured IQ Sample Pairs e.g. -n 2048, default = 2048"
            print "-d   Analyser Display On or Off e.g. -d ON or -d OFF, default = ON"
        else:
            # Process input args
            count = 0
 
            ipaddress = ""
            outfile = ""
            readtype = ""
            rbw = ""
            samprate = ""
            trigger = ""
            nosamples = ""
            dispupdate = ""
            vdebug = ""
 
            while count < len(argv):
                x = argv[count]
                if x == "-i":
                    ipaddress = argv[count+1]
                elif x == "-f":
                    outfile = argv[count+1]
                elif x == "-b":
                    readtype = argv[count+1].upper()
                elif x == "-r":
                    rbw = argv[count+1]
                elif x == "-t":
                    trigger = argv[count+1].upper()
                elif x == "-a":
                    samprate = argv[count+1]
                elif x == "-n":
                    nosamples = argv[count+1]
                elif x == "-d":
                    dispupdate = argv[count+1].upper()
                count += 1
 
            if ipaddress == "": ipaddress = "192.168.1.2"
            if readtype == "": readtype = "ASCII"
            if outfile == "": outfile = "iqcapture.txt"
            if rbw == "": rbw = "10MHz"
            if samprate == "": samprate = "10MHz"
            if trigger == "": trigger = "IMM"
            if nosamples == "": nosamples = "2048"
            if dispupdate == "": dispupdate = "ON"
 
            # Create an instance of the class and connect to the instruments
            fsx = scpicontrol.scpicontrol(timeout)
            fsx.connect(ipaddress,int(port))
 
            if fsx.connected == True:
                res = fsx.ask(debug,"*IDN?")
                if res != False:
                    print "Instrument Connected: " + res
                    print "RBW is: " + rbw
                    print "Trigger setting is: " + trigger
                    print "Sample Rate is: " + samprate
                    print "No. Samples requested: " + format(int(nosamples),",d")
                    print "Analyser Display is: " + dispupdate
 
                # Update the screen
                fsx.write(debug,"SYST:DISP:UPD " + dispupdate)
 
                # Continous sweep off so we can do synchronous reads
                fsx.write(debug,"INIT:CONT OFF")
 
                # Turn IQ capture ON
                fsx.write(debug,"TRAC:IQ:STAT ON")
 
                # Set RBW, Sample Rate, Trigger, tigger slope, pre-trig samples, no.samples
                fsx.write(debug,"TRAC:IQ:SET NORM,"+rbw+","+samprate+","+trigger+",POS,0,"+nosamples)
 
                # Get the IQ data
                # Create new IQReader Object
                iqr = iqreader.iqreader(fsx.handle)
                if readtype == "BIN":
                    fsx.write(debug,"FORM REAL,32")
                    fsx.write(debug,"TRACe:IQ:DATA:FORMat IQPair")
                    fsx.write(debug,"TRAC:IQ:DATA?")
                    iqr.biniqread(nosamples,outfile) 	# Read as binary
                elif readtype == "ASCII":
                    fsx.write(debug,"FORM ASCii")
                    fsx.write(debug,"TRACe:IQ:DATA:FORMat IQPair")
                    fsx.write(debug,"TRAC:IQ:DATA?")
                    iqr.asciiiqread(outfile)			# Read as ASCII
                    print "IQ saved to: " + outfile
 
                # Disconnect and close the socket
                if fsx.connected == True: fsx.disconnect()
    else:
        print "No Arguments Passed"
 
# Boilerplate syntax, call the main function
# Seems to significantly speed up script execution
if __name__ == "__main__":
	# Save the start time
    startTime = datetime.now()
    main()
    # Print script runtime
    print "Script Run Time: " + str(datetime.now()-startTime)

scpicontrol.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

import socket           # Get the socket functions
 
class scpicontrol:
    """Class to control instruments via SCPI over a socket with error
     checking and use of multiple connections"""
 
    # Internal variable to track if connected
    connected = False
 
    def __init__(self, timeout):
        """Initiate the Class using a timeout value for the socket
        and create an internal handle for the socket connection"""
 
        socket.setdefaulttimeout(timeout)
        self.handle = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 
    def connect(self,ip,port):
        """Connect to the instrument on IP and PORT"""
 
        try:
            self.handle.connect((ip,port))
            self.connected = True
            print "Connected OK to " + ip
            return True # Return OK if connected
        except:
            self.connected = False
            self.retry_connection(ip,port)
            return False
 
    def disconnect(self):
        """Disconnect from the instrument"""
 
        try:
            self.handle.close()
            self.connected = False
            print "Disconnected from Instrument"
            return True
        except:
            self.connected = False
            print "Error:  Failed to disconnect from instrument"
            return False
 
    def write(self,debug,msg):
        """Write SCPI message to instrument as ASCII string"""
 
        msg1 = msg+"\n"
        try:
            totalsent = 0
            while totalsent < len(msg1):
                sent = self.handle.send(msg1[totalsent:])
                if sent == 0:
                    raise RuntimeError("Socket Connection Broken")
                totalsent = totalsent + sent
            if debug == "ON":
                print "Message " + msg + " sent ok"
                result = self.ask(debug,"syst:err?")
                print ("Result: " + result)
            return True
        except:
            print "Error: Could not send the message " + msg + " to instrument..."
            self.connected = False
            return False
 
    def read(self):
        """Read data from instrument response as ASCII"""
 
        try:
            msg = ""
            finished = False
            while finished != True:
                chunk = self.handle.recv(1)
                if chunk == "\n":
                    finished = True
                if chunk == "":
                    raise RuntimeError("Socket Connection Broken")
                msg = msg + chunk.strip()
            return msg
        except:
            print "Error:  No answer from the instrument..."
            self.connected = False
            return False
 
    def ask(self,debug,msg):
        """Write command and read response from instrument"""
 
        if self.write(debug,msg) == True:
            res = self.read()
            if res != False:
                return res
 
    def retry_connection(self,ip,port):
        """Retry to connect to instrument if a problem occurs"""
 
        print "Not able to connect to " + ip
        res = raw_input("Retry to connect Y/n?\n")
        if res == "Y" or res == "y" or res == "":
            self.connect(ip,port)
        else:
            print ("OK, no problem.")

iqreader.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85

class iqreader:
    """Class to hande IQ reading from an R&S analyser in
       ASCII or Binary format"""
 
    def __init__(self,handle):
        """Initiate the Class"""
        self.handle = handle
 
    def asciiiqread(self,filename):
        """Read response as ASCII from instrument"""
 
        try:
            iqdata = ""
            finished = False
            asciifile = open (filename, "w")
            while finished != True:
                chunk = self.handle.recv(1)
                iqdata = iqdata + chunk.strip()
                if chunk == "\n":
                    finished = True
 
            iqdata = self.format_ascii_iqdata(iqdata)
            asciifile.write(iqdata)
            asciifile.close()
            return True
        except:
            print "Error:  No answer from the instrument..."
            self.connected = False
            return False
 
    def format_ascii_iqdata(self,iqdata):
        # split the iqdata on comma
        datalist = iqdata.split(",")
        samples = len(datalist)/2
        arrdata = ""
 
        counter = 0
        while counter < (samples*2):
            arrdata = arrdata + datalist[counter] + "," + datalist[counter+1] + "\n"
            counter = counter + 2
        return arrdata
 
    def biniqread(self,samples,filename):
        """Read response as BINARY from instrument"""
 
        try:
            # Read the first byte # from buffer
            self.handle.recv(1) 
 
            # Read num of digits to get for No of floats
            if int(samples) < 125000000:
                digits = self.handle.recv(1)
            else:
                self.handle.recv(1)
                digits = "10"
 
            # Read in how many bytes we have to get
            i = 0
            totalbytes = ""
            while i < int(digits):
                totalbytes = totalbytes + self.handle.recv(1)
                i += 1
 
            # Now read in the IQ data from the analyser memory
            # Write it to file as recieved
            binfile = open (filename, "wb")
            totbytes = int(totalbytes)
            nofloats = int(totalbytes)/4
            nosamples = int(totalbytes)/8
            buf = ""
            sampoutput = nosamples / 10
            i = 0
            count = 0
            while i = sampoutput:
                    print "Samples Read: " + format(i,",d")
                    count = 0
                count += 1
                i += 1
 
            binfile.close()
            return True
        except:
            print "Error: No answer from the instrument..."
            self.connected = False
            return False

Posted in Test and Measurement, Tutorials

Post Tags: » »

Test and Measurement Instrument control using PHP

Sunday, December 11th, 2011

There are various ways of remote controlling instruments such as remote desktop or VNC for getting access to the screen but often you want it to do something specific in an automated way. Many people use tools such as Labiew or even full on programming languages like Visual C, C#, VB etc with drivers (VXI or LXI) to achieve robust production solutions. Often when I am asked by customers to assist with some remote control, I turn to a web scripting language called PHP. Its not the normal scripting language people think of for instrument remote control but it has some benefits for me:

  1. It has socket support allowing control to port 5025 by sending SCPI commands as ASCII to write commands and read results.
  2. Its fast to develop, is very much like C but with simplicity.
  3. Has built in support for Regular Expressions and Text Parsing meaning string manipulation is quick and easy.
  4. As I’m familiar with the language I can knock up scripts quickly and send the SCPI command sequence to the customer.
  5. It can be used with a web server. There are possibilities with PHP to build a scaleable instrument remote control solution that is becomes a web application, meaning easy control of instruments just by logging onto a website.
  6. If required a complex system could be developed to easily store and recall results from a MySQL back end.

Of course there are other languages that could be used and are often used by customers such as Perl or Python to achieve the above tasks, I just happen to get on with PHP.

A simple example might be something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

// Allow the script to hang around waiting for connections
set_time_limit (0);
 
$fsq_ip_address = "192.168.1.2";
$debug = "OFF"
$port = 5025; 
 
// open socket to instrument
$fsq= fsockopen ($fsq_ip_address, $port, $errno, $errstr, 10); 
 
// open the output file for results
$fp = fopen ("output.txt", "w+"); 
 
// query instrument for its details and output result
writescpi($fsq,$debug, "*IDN?");
$result = readscpi($fsq);
logger("Instrument: " . $result); 
 
// close sockets and file
fclose ($fsq);
fclose ($fp); 
 
// Function Definitions
// Write string to the socket (output errors if debug enabled)
function writescpi($inst,$debug,$msg)
{
  fputs ($inst, $msg . "\n");
 
  if ($debug == true)
   {
    print ("\nCmd: " . $msg);
    fputs ($inst, "syst:err?\n");
    $result = readscpi($inst);
    print ("Result: " . $result . "\n");
  }
} 
 
// Read string from the socket
function readscpi($inst)
{
  $buf = fgets ($inst,100000);
  $buf = trim ($buf);
  return $buf;
} 
 
// output to screen and to file at the same time
function logger($result)
{
  global $fp;
  print ($result . "\n");
  fwrite ($fp, $result);
}

Posted in Test and Measurement, Tutorials

Post Tags: » » »