This tutorial describes how to use pbLua to do datalogging. You would use a datalog to collect data from sensors on your robot for later analysis. You can also use the collected data for calibration curves. We’ll even show you how to automatically start or select a datalogging application without connecting console when you start the NXT. That might come in handy if you’re running the NXT where it’s impractical or even impossible to connect a console…
Because we’ll be doing a lot of work with files, it would be a really good idea to review the pbLua FLASH File API to make sure you understand all of the features available for datalogging.
We’ll also be using the standard Lua string API, which has a lot of functions to deal with building a string out of individual bytes, and packing strings with binary data such as floating point numbers.
As a reminder, there is an excellent online version of Programming in Lua that is a complete description of Lua along with many programming examples, and an even better idea is to purchase the latest Programming in Lua book .
By the end of this tutorial, you’ll be able to use the FLASH file system in lots of different applications that require high speed permanent data storage.
- Storing Data In Strings
- Compacting Data In Strings
- Saving Strings In A File
- Start Your Datalogger Automatically
Before we store data in files, it would be a good idea to learn how to pack data efficiently in strings. Let’s say we want to store readings from an analog sensor like the HiTechnic Gyro.
Besides the raw analog data, it would be a good idea to store the timestamp along with each sensor reading. Here’s a simple example program that reads analog data from the gyro and returns it along with a timestamp until the orange button on the NXT is pressed:
-- Read gyro sensor until the orange button is pressed function GyroRead(port) -- Set up the gyro sensor with low sensitivity nxt.InputSetType(port,0) nxt.InputSetState(port,0,0) nxt.InputSetDir(port,1,1) -- Now start reading the sensor and putting out data repeat print( nxt.TimerRead(), nxt.InputGetStatus(port) ) until( 8 == nxt.ButtonRead() ) end -- And using the function - press the orange button on the NXT to stop it GyroRead(1)
And some typical results for a stationary gyro sensor that has had a bit of time to warm up:
52017 617 0 0 52019 615 0 0 52021 617 0 0 52023 616 0 0 52025 616 0 0 52027 617 0 0 52029 616 0 0
The values in each line correspond to:
- The timestamp just before the reading is taken
- The current analog value of the sensor measurement
- The state of I/O bit 0
- The state of I/O bit 1
The analog values have up to 10 bits of resolution, so they can range from 0 to 1023. The value of 616 is typical for the gyro at rest.
If you’re interested, you can count the characters, but it looks like if we store just the timestamp and the raw analog data it will take about 11 bytes to store the string. If you were to run this program later, it might take even more bytes to hold the string depending on the timestamp.
Let’s assume a worst case of 16 bytes per line of data. If we allocate 16K to a data file, that’s only 1,000 readings that we can store. We can actually allocate up to 64K for a single pbLua data file, in case you’re curious.
If we take a reading 10 times a second on a really interesting ride, like a roller coaster, that’s about 100 seconds of data in 16K.
We’re going to have to find some ways to compact our raw data strings, and fortunately it’s easy and quick to do this with pbLua.
The standard Lua string API does not provide a way to convert a full sized 32 bit number to its equivalent individual bytes. The pbLua API, however, has these functions:
|s = nxt.istr(n)||Accepts the integer given by n and returns a 4 byte string in native byte order.|
|n = nxt.stri(s)||Accepts the 4 byte string s in native byte order and returns the integer value that it represents.|
|s = nxt.fstr(f)||Accepts the float value given by f and returns a 4 byte string in native byte order.|
|n = nxt.strf(s)||Accepts the 4 byte string s in native byte order and returns the float value that it represents.|
What do we mean by native byte order? If you were to take the hex value 0x12345678 and convert it, the result would be 0x78563412. That’s because the ARM7 inside the NXT is a “little-endian” processor and we read numbers using “big endian” notation. For more information on this subject, read this article on endianness.
OK, so getting back on track, it’s clear we can easily convert an integer or a float to a series of bytes. To save our gyro reading and timestamp, it’s as easy as this:
-- Lots of tricks in this function, note the concatenation operator ".." and -- that the last two results from nxt.InputGetStatus() are simply discarded! function sampleString(port) return nxt.istr(nxt.TimerRead()) .. nxt.istr(nxt.InputGetStatus(port)) end
And if you follow along with the next results, you’ll see more examples of what the native little-endian numbers look like, and how easy it is to convert between numbers and byte strings.
-- Let's do a few examples first... > =string.byte(nxt.istr(0),1,4) 0 0 0 0 > =string.byte(nxt.istr(255),1,4) 255 0 0 0 > =string.byte(nxt.istr(65535),1,4) 255 255 0 0 > =string.byte(nxt.istr(-1),1,4) 255 255 255 255 -- And now use the function to generate the string... > port = 1 > s = sampleString(port) -- And dump the results. Can you figure out what the values were? > =string.byte(s,1,8) 148 26 10 0 106 2 0 0 -- It's easy if you do this... > -- The timestamp is the first 4 bytes (1 to 4) > =nxt.stri(string.sub(s,1,4)) 662164 -- The value is the second 4 bytes (5 to 8) > =nxt.stri(string.sub(s,5,8)) 618
Now that we know how to compact numnbers into raw bytes, and how to get the raw numbers back again later, we can move on to creating datalogs in the pbLua Flash File System.
If you have not yet looked over the pbLua FLASH File API then it might be a good idea to do that now. To store data in a file with pbLua we’ll need the following information:
- The name of the file to store the data in
- How much data we want to store
- A source of data to store
Recall that the pbLua Flash File System is not quite the same as to one on your computer. It’s simple minded, and needs to know how much data you’ll be trying to store so that it can find a block of unused FLASH big enough for your file. It’s better to find out that there’s not enough space beofre you start writing data.
Remember that you’ll want a unique name for your data file. If you want to use the same file name every time, then you’ll need to erase the old file before you start.
Just for fun, let’s write a little function that reads data from an analog port and writes it as fast as possible. If we timestamp every write, we’ll have an idea of how quickly we can write data to the filesysytem. If we save a timestamp (4 bytes) and data (4 bytes) we’ll need 8 bytes times 1000 records or 8000 bytes in our file.
-- Function to save 100 timestamped samples to a file as quickly as -- possible function save100 (port) -- Set up the gyro sensor with low sensitivity nxt.InputSetType(port,0) nxt.InputSetState(port,0,0) nxt.InputSetDir(port,1,1) -- Create an 800 byte file, save the handle local file = nxt.FileCreate("sampleFile", 800) -- And now read and save 100 individual samples and timestamps for i=1,100 do nxt.FileWrite( file, nxt.istr(nxt.TimerRead()) .. nxt.istr(nxt.InputGetStatus(port)) ) end -- And close the file nxt.FileClose(file) end -- And run the function on port 1 to try it out... save100(1)
Of course, now that you’ve written all the data to a file, it would be nice to read it back, so we’ll use the dumping function in this next example to do it, and I’ve even pasted the first few and last few records to see how long it took…
-- Function to dump timestamped samples from a file function DataDumper() -- Open the file file = nxt.FileOpen( "sampleFile" ) -- And now read individual samples and timestamps 8 bytes -- at a time until there are no more repeat local s = nxt.FileRead( file, 8 ) if s then local t = nxt.stri(string.sub(s,1,4)) local v = nxt.stri(string.sub(s,5,8)) print( string.format( "T:%08i V:%04i", t, v ) ) end until nil == s -- And close the file nxt.FileClose(file) end -- And run the function to see how fast the writing went... DataDumper() T:00950217 V:0617 T:00950221 V:0616 T:00950224 V:0619 ... T:00950574 V:0616 T:00950578 V:0615 T:00950581 V:0618 -- Yep, that's 100 samples written in 364 msec!
WOW! That’s reading 100 timestamps and cycles and writing the results in 364 msec – almost 300 Hz!
If you’ve read some of the other tutorials, you’ll know that the last section is usually the place where we put together a bunch of stuff we’ve learned in the previous section. This is no different. We’ll modify the basic datalogger to handle the case of erasing the file if it already exists. We’ll also let it use 8,000 bytes for 1000 entries, and finally, we’ll log every 100 msec or 10 times per second. This gives us about 100 seconds of data.
We’ll add a few other nice touches, such as a real-time update of progress on the LCD and the ability to stop logging and turn off the NXT when the user presses the orange button or when the end of the file is reached. Here’s what the new program looks like:
-- Complete data logger example s = [[ function DataLogger () -- Set up the gyro sensor on port 1 with low sensitivity nxt.InputSetType(1,0) nxt.InputSetState(1,0,0) nxt.InputSetDir(1,1,1) -- Check to see if the file exists, and if so, erase it... if nxt.FileExists("sampleFile") then nxt.FileDelete("sampleFile") end -- Create an 8000 byte file, save the handle local file = nxt.FileCreate("sampleFile", 8000) -- Clear the display nxt.DisplayClear() -- And now read and save up to 100 individual samples and timestamps for i=1,1000 do nxt.FileWrite( file, nxt.istr(nxt.TimerRead()) .. nxt.istr(nxt.InputGetStatus(1)) ) -- Update the LCD so we can see what's going on... nxt.DisplayText( i ) -- break out of the loop if the user hits the orange button if 8 == nxt.ButtonRead() then break end end -- And close the file nxt.FileClose(file) -- And turn off the NXT nxt.PowerDown() end -- Don't forget tot execute it :-) DataLogger() ]]
What’s going on here? What’s with the text inside the “[[" "]]” characters? It’s called a literal string, and you can read more about them here. We’ll be using this string to load up our file with the actual code.
WARNING: Always, always, always check and recheck your code before comitting it to FLASH. Then check it again. Especially if you’re expecting the code to work when the NXT is started automatically. You can always override things if you find a bug by connecting to the NXT with a USB cable. It will drop you to the console where you can fix your problem.
So, we’ve got the function we want to put into FLASH into a string called s. First, figure out how long the string is:
> =string.len(s) 970
To be safe, we’ll let the file system know that we want to store 1,024 bytes, so let’s make the startup file. Remember to erase the old one if you already have one.
> f=nxt.FileCreate("pbLuaStartup", 1024) > nxt.FileWrite(f,s) > nxt.FileClose(f)
That’s all there is to it! When you cycle power to the NXT, you’ll see that the datalogger will start automatically!
Now, before you get all excited, there is a big issue lurking in the Datalogger() code. Think about it for a while before you read the next paragraph…
The main problem I see is that if you have erased lots of files, you may run out of file descriptors when creating the sampleFile. To avoid this, make sure you do a FileFormat(0) before you commit your NXT to an expensive experiment. Make sure you don’t do a FileFormat(1) or you'll erase all of the files, including the startup file.
OK, now go out and do some datalogging!