Special Note
Since I first published this tutorial, I’ve improved the performance of the CRC algorithm and changed how the USB output is buffered. This makes the rate pretty much sustained at 3 msec intervals, with no gaps – great for high-speed datalogging!
Introduction
This tutorial describes how to exchange large quantities of data with the NXT using the XMODEM protocol. Now that we have FTP and HTTP protocols to do the busy work of fetching files for us, XMODEM is hardly used any more. It is a very primitive, minimal protocol for data exchange that is easy to understand, and easy to implement.
In order to use XMODEM, you’ll need a terminal program on your host computer that understands how to use XMODEM with CRC-16 checksums. On Windows machines, that’s Hyperterminal. On Linux and OSX machines that’s screen or minicom with a shell escape to rx/sz.
As a reminder, there is an excellent online version of Programming in Lua that is a complete description of Lua along with many programming examples, and an even better idea is to purchase the latest Programming in Lua book .
By the end of this tutorial, you’ll know how to use XMODEM to dump the entire contents of the NXT FLASH, or to load up a table with specific data. You’ll even learn how to do extremely high-speed data-logging by sending the data off the NXT at up to 8,000 bytes per second.
Contents
XMODEM Basics for pbLua
Traditional XMODEM implementations are what I’ll call blocking. You call the XMODEM send or receive function with a filename and it does not return until the file transfer is complete or it times out. For embedded systems like the NXT, this is not terribly useful because it means you can’t do anything like driving motors in response to sensor inputs while you’re transferring data. It also means you have to collect all your data in a file before streaming it off the NXT.
To get around these problems, I changed the way that XMODEM is used in an application. The XMODEM send and receive functions on the NXT are called in a loop and return information about the state of the protocol. The remote machine still sees a standard XMODEM data stream.
To make the XMODEMSend() and XMODEMRecv() functions easy to use, they operate similarly. Let’s start with the XMODEMRecv() function and then discuss XMODEMSend() afterwards.
Receiving Data From The Host
You can use the XMODEMRecv() function for accepting relatively large quantities of data from the host machine. One example might be restoring your FLASH file system from a backup after a firmware update, or initializing a large table.
Because pbLua’s XMODEM recieving function returns the individual 128 byte chunks as they arrive, you don’t need to collect the packets into a file before using them. You can actually store or process them as they arrive.
The first time you call nxt.XMODEMRecv() you’ll get a false reply to indicate that the initialization worked. If there is actually a packet ready to receive (unlikely) the reply will be a string. If the initialization failed or if the XMODEM machine is currently in the sending state, then the reply will be nil.
Subsequent calls to nxt.XMODEMRecv() return a string if there is a packet ready, and false if there is no packet. A packet must be stored or processed before the next call to nxt.XMODEMRecv().
When the host stops sending data to the NXT, or the protocol times out, or the user cancels the transfer on the host, the nxt.XMODEMRecv() function returns nil.
If you want the NXT to cancel the transfer, then call nxt.XMODEMRecv( nil ). The nil parameter indicates that the NXT does not want any more data.
To send a file from your host machine to the NXT, start up your terminal emulator and load the following example code into your NXT:
nxt.DisplayClear()
i = 0
repeat
s=nxt.XMODEMRecv()
if s then
nxt.DisplayText( string.format( "Record %i", i ) )
i = i + 1
end
until s == nil
You’ll see some upper case C characters on your terminal, which is just the XMODEM protocol on the NXT saying that it’s ready to receive a packet using CRC-16 checksums. You have about 15 seconds to start the file transfer before the NXT gives up on the receiving side.
Now start your XMODEM file transfer from the terminal emulator on your host and pick a file to send – it can be any size, but at a transfer rate of about 8 kBytes per second a very large file might take a looooong time.
You should see “Record xxx” in the lower line with xxx incrementing every time a new packet of 128 bytes is received by the NXT.
That’s not terribly interesting, so let’s try something else. Let’s transfer a small text file – less than 12K or so to the NXT and store it in a table. Recall that tables can store any type of data, using any value as a key.
In this application, we’ll use numbers starting from 1 to index individual 128 byte strings in the table that will hold the text file. The standard Lua table.insert() comes in handy to do this without requiring us to keep a local counter for the current record number.
Another advantage of using a table is that the data in the table is not contiguous. That means that we don’t need to find a free block of RAM 12K in size to store the entire file We just need anough 128 byte blocks of RAM to hold the file.
Here’s the sample code, and I’ll assume that you have a file of about 12K in size handy to send to the NXT. It can be any data at all, but text is easier to work with and display.
nxt.DisplayClear()
a={}
repeat
s=nxt.XMODEMRecv()
if s then
table.insert(a,s)
end
until s == nil
OK, now, you’ve got a table a in the NXT that has the entire file contents. Now let’s echo them back out to the console to see what we’ve got. Recall that nxt.FileWrite() with a handle of 0 sends the data out to the current console without a terminating line end.
for _,s in ipairs(a) do nxt.FileWrite(0,s) end
How cool is that! Just remember that if your original file does not have CRLF line endings it might end up looking a bit goofy in Hyperterminal. It’s pretty easy to extend this simple code framework in the first two examples to do some useful work with the received packet whenever the XMODEMRecv()
function is called.
Sending Data To The Host
You can use the XMODEMSend() function for transmitting relatively large quantities of data to the host machine. One example might be backing up your FLASH file system, or logging a large quantity of readings to the PC.
Because pbLua’s XMODEM sending function tells you when it is ready to accept the next block of data to send to the host, you don’t need to collect all the data into a file first. You can actually generate them as needed.
The first time you call nxt.XMODEMSend() you’ll get a false reply to indicate that the initialization worked. If the initialization failed or if the XMODEM machine is currently in the receiving state, then the reply will be nil.
Subsequent calls to nxt.XMODEMSend() return a true if it is ready for a new packet, and false if it’s still sending the previous packet. A packet must be generated before the next call to nxt.XMODEMSend(). Only the first 128 bytes of a string passed to nxt.XMODEMSend() are used. If the string is shorter than 128 bytes, it is padded out with NULL characters.
When the protocol times out, or the user cancels the transfer on the host, the nxt.XMODEMSend() function returns nil.
If you want the NXT to cancel the transfer, then call nxt.XMODEMSend( nil ). The nil parameter indicates that the NXT does not have any more data.
Let’s start with a simple example. Send 256 blocks of data to the host machine. Each block is a string of 128 characters that correspond to the block number.
To recieve a file from the NXT on your host machine, start up your terminal emulator and load the following example code into your NXT:
p = ""
i = 0
repeat
s = nxt.XMODEMSend(p)
if( s ) then
if i == 256 then
p = nil
else
p = string.rep( string.char(i), 128 )
i = i + 1
end
end
until s == nil
Now start your XMODEM file transfer from the terminal emulator on your host and specify a file name to recieve the data to. It should take about 4 seconds to transfer all 32K of the data to your host machine. You can open the file with a text editor capable of displaying binary data to look at the data.
Let’s try another example – this time we’ll dump the entire contents of the FLASH memory of the NXT to a file!
p = ""
addr = 0x100000
repeat
local s = nxt.XMODEMSend(p)
if( s ) then
if addr >= 0x140000 then
p = nil
else
p = nxt.MemRead(addr,128)
addr = addr + 128
end
end
until s == nil
Again, start the XMODEM receive function and specify a name for the file. On my test machine, I managed to get over 24KB per second transferred from the NXT to the host!
This brings up a very neat application, namely logging data to a file right on your PC using XMODEM data transfer.
HighSpeed Datalogging
As a demonstration of how we can use XMODEM data transfer for very hight speed datalogging, let’s connect the light sensor to the NXT and log data as fast as we possibly can.
For this experiment, we’ll sample the light sensor and print its value along with the current timestamp to a string. When the string is more than 128 bytes long, we’ll fire it off to the XMODEM protocol, lopp off the the first 128 bytes and then generate more data until we have 128 bytes again.
We’ll do this for 10 seconds or so. Note that this is not the most effective way to do things. A future tutorial will address this producer/consumer issue and present a solution using coroutines – the build in multi-tasking feature of Lua.
Here’s the sample code:
-- Function to set up the light sensor and then start transferring data
-- to the host as quickly as possible. Call the function, then start the
-- XMOPDEM transfer on the host, and then press the orange button to
-- start the sampling...
function LogLight( port )
-- Set up the light sensor on port 1 for passive mode
nxt.InputSetType(port,0)
nxt.InputSetState(port,0,0)
nxt.InputSetDir(port,1,1)
repeat
-- Wait here until the orange button is pressed
until( 8 == nxt.ButtonRead() )
local startTicks = nxt.TimerRead()
local sampleTick = startTicks
local p = ""
local d = ""
local s = true
repeat
local nowTicks = nxt.TimerRead()
-- Grab a light sensor sample and add it to our string if
-- at least 3 milliseconds have passed and we're still sending
if (sampleTick + 3 <= nowTicks) and (p ~= nil) then
d = d .. string.format( "%06i %06i\r\n", nowTicks, nxt.InputGetStatus(port) )
sampleTick = nowTicks
end
-- Only try sending if we have more than 128 bytes to send
if string.len(d) > 128 then
s = nxt.XMODEMSend(p)
if( s ) then
if startTicks + 10000 < nowTicks then
p = nil
else
p = string.sub(d,1,128)
d = string.sub(d,129)
print(p)
end
end
end
until s == nil
end
And here’s how to run it:
-- Start the high-speed datalogger LogLight(1)
The program will pause waiting for you to press the orange button, but before you do, start the XMODEM receive to a file. The NXT will fire light sensor data at the host machine as quickly as it can for 10 seconds.
If you edit the file, you’ll see something like this:
119345 000623 119348 000619 119351 000621 119354 000623 119357 000619 119360 000623 119363 000622 119366 000622 119369 000623 119372 000623 119375 000619 119378 000619
Now that’s pretty fast datalogging! I’ll bet that you can think of a LOT of really interesting experiments that could be done with pbLua in a high-school physics lab.
