Test Gear Triumph! (Arduino to the Rescue)

Tidying up my desk a bit yesterday, I found a circuit on a breadboard I’d left hanging. Months ago I was looking into notch filters for removing mains hum, to clean up a ELF/VLF signal a wee bit. I’d put together a bootstrapped twin-T notch filter, but had got rather frustrated when testing it. I wanted to get a general idea of its response and (assuming it looked ok) tune it to 50Hz.

But I’ve only got a USB port Bitscope oscilloscope (the BS10 mixed-signal model) which does do basic frequency analysis and even has a signal generator built in. Unfortunately there’s no sweep for the sig gen, and the UI is so clunky I wound up making a little generator with an easily-twiddled knob. That still didn’t really give me what I was after in being able to clearly see what was going on.

Anyhow, today I thought I’d take another look. Got everything set up, did some manual sweeping which showed that the component values I’d used were quite a way out (more like 70Hz). But still no clear visualisation of the overall response.

Staring at the desk, pondering what to do next…there’s an Arduino Uno right in front of me. I’ve spent a fair while getting to know the things over the past few weeks. I’d noticed in passing that it had a tone() method, but hadn’t actually played with it. Ok, about 15 minutes later I had this loaded:

void setup (){
}

void loop() {
  int i;

  for (i = 35; i <= 100; i++) {
    tone(11, i);
    delay(10);
  }
}

A sweep generator!

Ok,  its frequency range is limited and it gives a square wave out. So I took the output from pin 11 and fed that to a simple RC filter (15k, 220nF) which took the buzziness down a bit. Stray harmonics aren’t that much of an issue for the current problem, and 35-100 Hz cover the range I’m looking at.

One thing the Bitscope’s waveform generator allows is the fairly accurate setting of frequency. So I set that at 50Hz and put it in one scope input, the output of my notch filter into the other. After a bit of fiddling to get levels reasonably stable, I got this:

notch

The yellow is my 50Hz reference, green the notch filter response. The harmonics on the ref are pretty dire – dunno, I guess it must be clipping. But look at that lovely notch in the green! Around 70 or so Hz, as measured before.

So this setup can help me quickly tune the notch down to where it’s needed. But that isn’t the real triumph here. What I wasn’t sure about is the rest of the response of the active notch. Where the passive notch goes from flat into a 6dB (I think) / octave drop into the notch, this version has noticeable mounds either side. Those are potentially very undesirable. If you look at the 50Hz marker here, my filter as it stands would boost that frequency. While I’m sure I can get the notch in a much better position than this, any drift (maybe due to environmental factors) could be very bad. So at the cost of less sharp notch, I reckon on balance the passive version is probably the one to go for.

PS.

A few hours on, and a bit more progress. I pulled out the active notch circuit, did calculations again and plugged in a passive one. Well, I say passive, am using a TL074 to buffer the signal.

The basic filter circuit is this:

rc_twin_t

Fc = 1/(2 pi R C)

Using C = 100nF (2C just two of them in parallel) and 33k for each of the two Rs on top, a single 15k for the R/2 I got something looking like a cleanish notch, centred on 47.8Hz. It took a little trial & error. The capacitors are just off-the shelf, ceramic I think, probably 10% tolerance but came from the same batch so should be reasonable well matched. 1% resistors, again off-the shelf, same batch.

I forgot to take a screenshot…

But as I measured previously, the ambient mains hum here also contains a significant amount of 3rd harmonic, ie. 150Hz. So I did the sums again for this.

Ran into a slight snag with my setup though – when sweeping up through a reasonable range for it to go over the 150Hz target, the spectrogram display was all over the place.

But, as an alternative to sweep, you can also test freq response with white noise (or an impulse, but that’s another story). Coincidentally I was playing with a pseudorandom number generator just yesterday (for DOG-1), so knew what to look for. I found one, to which I’ve made minor tweaks –

#define speakerPin 11

unsigned long lastClick;

void setup() {
  // put your setup code here, to run once:
   pinMode(speakerPin,OUTPUT);
   lastClick = micros();   
}


/* initialize with any 32 bit non-zero  unsigned long value. */
#define LFSR_INIT  0xfeedfaceUL
/* Choose bits 32, 30, 26, 24 from  http://arduino.stackexchange.com/a/6725/6628
 *  or 32, 22, 2, 1 from 
 *  http://www.xilinx.com/support/documentation/application_notes/xapp052.pdf
 *  or bits 32, 16, 3,2  or 0x80010006UL per http://users.ece.cmu.edu/~koopman/lfsr/index.html 
 *  and http://users.ece.cmu.edu/~koopman/lfsr/32.dat.gz
 */  
#define LFSR_MASK  ((unsigned long)( 1UL<<31 | 1UL <<15 | 1UL <<2 | 1UL <<1  )) unsigned int generateNoise(){    // See https://en.wikipedia.org/wiki/Linear_feedback_shift_register#Galois_LFSRs    static unsigned long int lfsr = LFSR_INIT;  /* 32 bit init, nonzero */    /* If the output bit is 1, apply toggle mask.                                     * The value has 1 at bits corresponding                                     * to taps, 0 elsewhere. */    if(lfsr & 1) { lfsr =  (lfsr >>1) ^ LFSR_MASK ; return(1);}
   else         { lfsr >>= 1;                      return(0);}
}


void loop() {
      /* ... */
      if ((micros() - lastClick) > 500 ) { // Changing this value changes the frequency.
        lastClick = micros();
        digitalWrite (speakerPin, generateNoise());
      }

}

One tweak to use pin 11 as I’d already got that wired up. The other is rather sweet. The original code had a loop delay of 50 micros, related to the bandwidth. But that again wasn’t very clear on the spectrogram. Was nice white noise, but I’m only interested in the low end here. Making the micros 500, and letting the display accumulate for a minute, produced this:

notch-2

There’s a nice notch pretty close to 50Hz, plus my new one, near enough at 150Hz (measured at 145Hz). The peak on the left is probably just an artifact of the setup – FFT does that sort of thing. Also the relative shallowness of the second notch I reckon is at least in part to the fact that it uses a linear scale on the spectrogram.

The values I used here were C = 47n, R = 22k, pleasingly standard values (the resistors gived those capacitors calculated at 22.57k, which was handy).

I’ve just got this set up on the breadboard around a TL074 quad op amp, using 3 op amps for unity gain buffers (each with a 1M to ground). Those things have input resistance of 10^12 ohms. So I’m now thinking I might just use one of them as the input stage for an ELV/VLF receiver. The 2N3819 input stage of the BBB-4 receiver I was going to try has a 10M resistor to ground, seems like plenty of leeway for that here. Input buffer, maybe give it variable gain of something like 1-100, to these filters (perhaps adding a little more gain along the way), then use the spare op amp to drive a couple of transistors for a small speaker/headphone level output.

Just trying it with a longish wire at the input, computer speakers at out, still way too much mains-derived noise to hear any natural signals, but the difference between the different stages of the circuit is really noticeable. I’ll have to get it soldered up, battery power, take it up the fields.

And try it when there’s a thunderstorm around 🙂

 

Advertisements

Arduino ESP8266 ‘Shiald’, step by step

A little while ago I ordered a couple of Arduino Uno cards along with a couple of ESP8266 WiFi shields. It being my first hands-on with Arduinos, I was rather naive in my choice of shields. I got mine from banggood.com when they were listed as “ESP8266 ESP-12E UART WIFI Wireless Shield TTL Converter For Arduino UNO R3 Mega“, but while figuring out how to use them I found that various other budget tech vendors sell them. Their identifying feature is a charming little typo printed on the PCB :

Arduino ESP8266 WiFi Shiald Version 1.0 by WangTongze

DSCN0002

The major problem is that the only official documentation is in Chinese (Mandarin?), something I haven’t a clue about. But by trawling the web and with a lot of trial and error I was eventually able to get code running on the card. I’ve written the process up spread across previous posts here, but it is rather convoluted, so for future ref. I’m pulling it together here. If you haven’t already bought one of these shields, you may well be better off getting something like a Wemos card.

Requirements

  • Arduino ESP8266 WiFi Shiald Version 1.0 by WangTongze
  • Computer with Arduino IDE loaded (I’m using a regular laptop with Ubuntu OS)
  • USB-TTL level serial converter – NB. I didn’t have one of these, but it turns out to be straightforward to use an Arduino Uno as a pass-through converter
  • USB cable, jumper leads (4 with a socket on one end)

Flashing Firmware

I must admit I don’t know if this step is entirely necessary, there may well be a quicker approach. But it worked for me, and is useful for resetting the card.

Using Arduino as USB-Serial Converter for the ‘Shiald’

The wiring is as follows :

Shiald Debug TX  => Uno Pin 1 (TX)
Shiald Debug RX  => Uno Pin 0 (RX)
Shiald Debug 5V  => Uno 5V
Shiald Debug GND => Uno GND
Uno Reset        => Uno GND

(Uno USB => Computer USB)

Flashing

Before connecting the Arduino to the computer, set the DIP switches on the Shiald as follows:

1 Off
2 Off
3 On
4 On

The following I got from the Wemos page Tutorial – Returning a Wemos D1 Mini to Factory Firmware (AT) :

I had to tweak my paths a little bit, I forget the details, but whatever it took to get esptool.py running from the shell.

The script needed tweaking for the appropriate paths. Run:

ls /dev/tty*

–  and the appropriate port should be obvious on the resulting list. My version of the script looks like this:

#!/bin/sh
# ESPToolDir="$HOME/Downloads/esptool"
FirmwareDir="$HOME/Arduino/ESP8266_NONOS_SDK"
cd "$FirmwareDir" 
port=/dev/ttyACM0
if [ ! -c $port ]; then
 port=/dev/ttyACM1
fi
if [ ! -c $port ]; then
 echo "No device appears to be plugged in. Stopping."
fi
printf "Writing AT firmware in 3..."
sleep 1; printf "2..."
sleep 1; printf "1..."
sleep 1; echo "done."
echo "Erasing the flash first"
"esptool.py" --port $port erase_flash

"esptool.py" --chip esp8266 --port $port \
 write_flash -fm dio -ff 20m -fs detect \
 0x0000 "$FirmwareDir/bin/boot_v1.7.bin" \
 0x01000 "$FirmwareDir/bin/at/512+512/user1.1024.new.2.bin" \
 0x3fc000 "$FirmwareDir/bin/esp_init_data_default_v08.bin" \
 0x7e000 "$FirmwareDir/bin/blank.bin" \
 0x3fe000 "$FirmwareDir/bin/blank.bin"

echo "Done."

The messages given by esptool.py are pleasingly informative, but I found I have to press the reset button on the Shiald when the message got to:

...
Hard resetting...
esptool.py v2.2.1
Connecting...

The original script suggested using miniterm to check this had worked. I used the Arduino IDE. First unplug the USB and set the DIP switchesto all Off.

After plugging back in again & launching the IDE, go to Tools -> Port and choose whatever looks right. Under Tools -> Board choose NodeMCU 1.0 (ESP 12E module). Then go to Tools -> Serial Monitor.

In the serial monitor, set the baud rate to 112500 and then click reset on the Shiald.

You should get a message that ends in ‘ok‘.

At this point you should be able to communicate with the Shiald using AT commands. Two useful things:

AT+UART_DEF=9600,8,1,0,0

This will flip the baud rate down to 9600.

AT+GMR

Gives the versions of various things.

At this point it should be possible to upload software to the Shiald (with the DIP switches Off, Off, On, On) from the Arduino IDE.

Tools -> Board NodeMCU 1.0 (ESP12)

 

I’ve found that it often takes several attempts (and hits of the reset switch) to get a successful upload, no matter what the baud rate.

e.g. this minimal web server:

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WebServer.h>
const char* ssid = "AllPay Danny";
const char* password = "not this";
ESP8266WebServer server(80); // HTTP server on port 80

IPAddress ip(192, 168, 0, 14); // where xx is the desired IP Address
IPAddress gateway(192, 168, 0, 1); // set gateway to match your network
IPAddress subnet(255, 255, 255, 0); // set subnet mask to match your network

void setup() {
 Serial.begin(9600); 
 WiFi.disconnect(); // Disconnect AP

 WiFi.config(ip, gateway, subnet);

 WiFi.mode(WIFI_STA); 
 WiFi.begin(ssid, password); // Connect to WIFI network
// Wait for connection
 while (WiFi.status() != WL_CONNECTED) {
 delay(500);
 Serial.println(".");
 }
 Serial.print("Connected to ");
 Serial.println(ssid);
 Serial.print("IP address: ");
 Serial.println(WiFi.localIP());
server.on("/", [](){
 server.send(200, "text/plain", "Hello World");
 });
server.begin(); // Start HTTP server
 Serial.println("HTTP server started.");
}
void loop() {
 server.handleClient();
}

Pointing a browser at the chosen IP address should now work.

There’s a complication to comms between the Arduino and the Shiald. It seems the serial Tx/Rx lines of the ESP8266 connect to ports 1 & 2 on the Shiald – the Arduino’s Tx/Rx.

Again, I’m not really sure how essential this is, but it certainly works to wire other ports on the Arduino to the serial on the Shiald and use the SoftwareSerial lib. For reliability a baud rate of 9600 seems advisible.

Here’s a little example that worked for me:

#include <SoftwareSerial.h>
SoftwareSerial mySerial(2, 3); // RX, TX on Arduino
void setup() {
 Serial.begin(9600);
 mySerial.begin(9600);
}
void loop() {
 if (mySerial.available()) {
 String msg = mySerial.readString();
 Serial.print("Data received: ");
 Serial.println(msg);
 }
}

Remember before uploading to remove the jumpers and change the board settings in the Arduino IDE.

At runtime, the following wiring worked for me:

Arduino   | Shiald

GND       - Debug GND
+5v       - Debug 5v
Digital 2 - Digital 0
Digital 3 - Digital 1

For my own application I want to be able to read values from the Analog Ins of the Arduino (6) and ESP8266 (1) and expose these (and one or two other little things) on a web server over Wifi. I’ve made a bit more progress towards this, will upload code to this project’s GitHub repo once I’ve tweaked to hide passwords.

So…this is my current setup:

DSCN0004

The lower part is a Shiald piggybacking an Arduino Uno, the upper another Arduino acting as a serial interface direct to the Shiald. Each Arduino is going to a USB port on the laptop.

The wiring – the Shiald has its pins 1 & 2 bent out of line and connected instead via jumpers to pins 2 & 3 on the host Arduino. The Arduino acting as a serial interface has is as above, less the power lines.

For writing code to the Shiald, the DIP switches are at Off, Off, On, On. At runtime they’re at On, On, Off, Off.

I’m still using the Arduino IDE, flipping between USB port and board (Arduino Uno/NodeMCU 1.0).

It’s a bit of a pain flipping between the configs, but takes less time than uploading a reasonably long program to the Shiald, so I can’t really complain.

 

 

ESP8266 Shiald Progress!

I’m really tired, but while trying to watch TV got to thinking about the Wifi board I’ve been playing with (described in previous post). I’d got as far as loading firmware that allowed it to speak AT codes. Couldn’t resist having a quick look at what could be done next. Luckily I went to my bookmarks first rather than looking at my notes here, because there was a page I must have bookmarked early on and forgotten about : Arduino UNO + ESP8266 ESP-12E UART WIFI Shield. It contains code for a minimal web server.

Looking at an image in this post reveals that the Shield there is there very same Shiald [sic] I have. Only problem, the author uses a USB-serial adapter to talk to it, something I don’t have. But wait – I found a way of rigging the Arduino to act as such an adapter (previous post).

I saw somewhere, and confirmed (by using a tablet to scan for WiFi networks) that the default IP address for the Shiald is rather an obscure one, off my local subnet anyway. But a bit of googling gave me the info necessary to set the IP to something else.

After fiddling a bit with the baud rate, a little blue light started flashing next to the ESP8266 chip, and it worked!

In the IDE:

Screenshot from 2018-02-07 23-07-57

In the Serial console:

Screenshot from 2018-02-07 23-11-12

Screenshot from 2018-02-07 23-11-35

And in a browser! Woo-hoo!

Screenshot from 2018-02-07 22-45-33

Here’s my tweaked version of the script:

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WebServer.h>
const char* ssid = "AllPay Danny";
const char* password = "not this";
ESP8266WebServer server(80); // HTTP server on port 80

IPAddress ip(192, 168, 0, 14); // where xx is the desired IP Address
IPAddress gateway(192, 168, 0, 1); // set gateway to match your network
IPAddress subnet(255, 255, 255, 0); // set subnet mask to match your network

void setup() {
 Serial.begin(115200); 
 WiFi.disconnect(); // Disconnect AP
 
 WiFi.config(ip, gateway, subnet);
 
 WiFi.mode(WIFI_STA); 
 WiFi.begin(ssid, password); // Connect to WIFI network
// Wait for connection
 while (WiFi.status() != WL_CONNECTED) {
 delay(500);
 Serial.println(".");
 }
 Serial.print("Connected to ");
 Serial.println(ssid);
 Serial.print("IP address: ");
 Serial.println(WiFi.localIP());
server.on("/", [](){
 server.send(200, "text/plain", "Hello World");
 });
server.begin(); // Start HTTP server
 Serial.println("HTTP server started.");
}
void loop() {
 server.handleClient();
}

The blog post goes on to Part 2 Upload code to Arduino, which I’ll try next – when I’m properly rested 🙂

PS.

Just tried this Part 2 bit, essentially comms between Shiald & Arduino. It nearly worked :

Data received: . .
Conoected to AllPay Danoy
IP address: 092.168.0.04C!⸮⸮⸮⸮ٕɁ⸮⸮

I’ve read somewhere that the Software Serial struggles at high baud rates, and this example is using 115200 so presumably that’s the problem. Bit of tweaking required.

PPS.

I flipped the baud rate in the code based on that in the blog post to 9600, and with the Arduino as serial converter uploaded the new code to the Shiald (at 115200 baud), set as NodeMCU 1.0. Uploading took a good few attempts, but finally it worked.

I also changed the Arduino part of the code to use different ports :

#include <SoftwareSerial.h>
SoftwareSerial mySerial(2, 3); // RX, TX on Arduino
void setup() {
 Serial.begin(9600);
 mySerial.begin(9600);
}
void loop() {
 if (mySerial.available()) {
 String msg = mySerial.readString();
 Serial.print("Data received: ");
 Serial.println(msg);
 }
}

The wiring I now have as :

Arduino   | Shiald

GND       - Debug GND
+5v       - Debug 5v
Digital 2 - Digital 0
Digital 3 - Digital 1

The switches on the Shiald are at (1,2,3,4) On, On, Off, Off.

And finally, the Hello World is still visible on the IP address I set. And what’s more, in the serial monitor (now set to 9600 baud) I see:

Connected to AllPay Danny
IP address: 192.168.0.14
HTTP server started.

Yay! It all works.

So…

Next

First thing I should do is pull together all the various bits from the last post and this, with relevant material from linked pages, and write it up as a from scratch to here procedure. I won’t remember, and also anyone that buys the same boards will stand a chance of getting things going.

Then I need to think about what I’m going to do on the analog/sensor side. What I can do with the hardware I’ve got is fairly limited – a key factor being the speed of the data acquisition on the Arduino. But I should have the necessary for me to build something that operates end-to-end with essentially the same topology as my target design.

Regarding code on the Arduino & Shiald, the next steps will be to :

  1. Get the data from the single Analog Input on the Shiald, buffer/filter it and expose it on a local web server. With a little analog pre-amp & filter this should be enough for a single-channel seismometer.
  2. Do the code necessary on a regular computer to access and do something with the data from the web server on the Shiald.
  3. Get the data from the 6 Analog Inputs on the Arduino, buffer/filter it, transfer it to the Shiald and again expose on a local web server. I might well try the analog bandpass filter idea mentioned in my previous post.
  4. As 2. but for the 6 channels.

A global job to put together in parallel with the above is the code necessary for self-description of the units to provide status information alongside the data. RDF and Web of Things time!

So now I’ve got fairly fun jobs to get on with on every side of this project :

  1. Sensor hardware
  2. Arduino/Shiald software
  3. Comms/post-processing software – I can get on with the Deep Learning bits using online sources, haven’t looked at that for weeks
  4. Notification system – hook the Deep Learning bit output to Twitter

I may have to get the dice out…

//// note to self

danny@lappie:/dev$ esptool.py --port ttyACM0 --baud 9600 flash_id
esptool.py v2.2.1
Connecting........_____....._____
Detecting chip type... ESP8266
Chip is ESP8266EX
Uploading stub...
Running stub...
Stub running...
Manufacturer: c8
Device: 4016
Detected flash size: 4MB
Hard resetting...

Arduino – initial experiences

skip to Arduino/WiFi bit, also Issues Raised and a Cunning Plan

Requirements & Constraints

On the hardware side of this project, I want to capture local seismic and ELF/VLF radio data. I’ve given myself two major constraints: it should be simple; it should be low cost. These constraints are somewhat conflicting. For example, on the seismic side, a simple approach would be to purchase a Raspberry Shake, an off-the-shelf device based on a Raspberry Pi and an (off-the-shelf) geophone. Unfortunately, these gadgets start at $375 USD, and that’s only for one dimension (and there may be software licensing issues). I want to capture 3D data, and want to keep the price comfortably under $100. Note that project non-constraints are absolute measurement, calibration etc. So the plan is to hack something. I’m taking rather a scattergun approach to the hardware – find as many approaches as are feasible and try them out.

Both the seismic and radio sensor subsystems have particular requirements when it comes to physical location. The seismic part should ideally be firmly attached to local bedrock; the radio part should be as far away as possible from interference – mains hum being the elephantine wasp in the room. For my own installation this will probably mean bolting the seismic part to my basement floor (which is largely on bedrock) and having the radio part as far up the fields as I can get it.

What seems the most straightforward starting point is to feed data from the sensors into a local ADC, pass this through a microcontroller into a WiFi transceiver, then pick this up on the home network. (WiFi range may well be an issue – but I’ll cross that bridge when I come to it).

The two microcontroller systems that seem most in the frame due to their relatively low cost are the aforementioned Raspberry Pi and the Arduino family. For a first pass, something Arduino-based seems the best bet – they are a lot cheaper than the Pis, and have the advantage of having multiple ADCs built in (compared to the Pi’s none – though there are straightforward add-ons).

Arduino Fun

Quite a while ago I ordered a couple of Arduino Unos and WiFi shields from Banggood, a China-based retailer of low cost stuff. My only prior experience with Arduinos was when my brother was building something MIDI-related and hit a code problem. He mailed me on the offchance and amusingly I was able to solve the problem in my reply – it was a fairly easy bit of C (I hadn’t done any other C for years, but coding is coding).

I instantly fell in love with the Arduino boards (actually a clone by GeekCreit). After very little time at all I was able to use the Arduino IDE to get some of the example code running on one of the devices. Light goes on, light goes off, light goes on… Very user friendly.

ESP8266 Nightmares

In my naivety, I assumed the WiFi shields would be as straightforward. Most probably are, but the ones I ordered have been distinctly painful so far. But I can at least put slow progress so far down as a learning experience. Essentially the ones I got have several issues. The story so far:

The boards I got are labeled “Arduino ESP8266 WiFi Shiald Version 1.0 by WangTongze”. Yup, that’s ‘Shiald’, not auspicious. The first major issue was that the only official documentation was in Chinese (mandarin?). I wasted a lot of time trying to treat them as more standard boards. But then found two extremely helpful blog posts by Claus : Using ESP8266 Shield ESP-12E (elecshop.ml) by WangTongze with an Arduino Uno and Arduino ESP8266 WiFi Shield (elecshop.ml) by WangTongze Comparison.

The first of these posts describes a nifty little setup, using an Arduino board as a converter from USB to TTL level RS232 that the Shiald can understand (I didn’t think to order such an adapter). It looks like this:

arduino1

By default the Shiald plugs its serial TX/RX pins to the Arduino’s, which does seem a design flaw. But this can (apparently) be flipped to using software serial via regular digital I/O pins on the Uno. A key thing needed is to tell the Shiald to use 9600 baud rather than its default 115200. The setup above allows this. This part worked for me.

However, at this point, after bending the TX/RX pins out of the way on the Shiald and plugging it in on top of the Uno (with jumpers to GPIO for TX/RX), I couldn’t talk to it. So going back to Claus’s post, he suggests updating the Shiald’s firmware. Following his links, I tried a couple, ended up with the setup spewing gibberish (at any baud rate).

At this point – after a good few hours yesterday, I was ready to cut my losses with the WiFi Shialds. I’d mentioned to danbri that I was struggling with these cards and he mentioned that he’d had the recommendation (from Libby) of Wemos cards. So I started having a look around at what they were. As it happens, they have a page on their wiki Tutorial – Returning a Wemos D1 Mini to Factory Firmware (AT). The D1 uses the same ESP8266 chips as my Shiald, so this morning, nothing to lose, adjusted the script and gave it a shot. Going back to the setup in the pic (with DIP switches tweaked as Claus suggests) it worked! (Tip – along the way of flashing, I had to press the Shiald’s reset button a couple of times).

arduino-at

So far so pleasing – I thought I might have bricked the board.

(See also ESP8266 Wifi With Arduino Uno and Nano)

After this I’d tried with the Shiald mounted on top of the Arduino in a good few configurations with various different software utilities, haven’t yet got everything talking to everything else, but this does feel like progress.

Issues Raised and a Cunning Plan

Sooo… these Shialds have been rather thieves of time, but it’s all learning, innit.

These bits of play have forced me into reading up on the Arduinos a bit more. For this project, a key factor is the ADC sample rate. It seems that the maximum achievable for a single ADC is around 9kHz (with 10 bit precision). That should be plenty for the seismic sensor. The radio sensor is another matter. I’d like to be able to cover up to say 20kHz, which means a sampling rate of at least 40kHz. I’m still thinking about this one, but one option would be to use an ADC shield – these ones from Mayhew Labs look plenty – though getting the fast data along to WiFi could well be an issue (intermediate baud rates). If necessary, some local processing could be a workaround. I have been intending to present the radio data to the neural network(s) as spectrograms so maybe eg. running an FFT on the Arduino may be feasible.

Along similar lines, I may have a Cunning Plan, that is to shift some of the processing from digital to analog. This is likely to need a fair amount of research & experimentation, but the practical circuitry could be very straightforward. It seems at least plausible that the earthquake precursors are going to occur largely in particular frequency regions. The Arduino has 6 analog inputs. So imagine the radio receiver being followed by 6 bandpass filters, each tuned around where precursors may be expected. A simple diode & (leaky) capacitor peak level detector for each of these could provide a very crude spectrogram, at a rate the Arduino could easily handle. Op amp BP filters are straightforward and cheap, so an extra $5 on the analog side might save $40 and a oad more work afterward.

Regarding the research – a key source is (of course) Renato Romero’s vlf.it, notably the OPERA project – although that does seem to focus at the low end of potential frequencies.

Preconditioning Seismic Data

The filtered data I have is CSV with lots of lines with the fields :

datetime, latitude, longitude, depth, magnitude

The latter 4 fields will slot in as they are, but a characteristic of seismic events is that they can occur at any time. Say today 4 events were detected at the following times:

E1 01:15:07 lat1 long1 d1 2.2
E2 01:18:06 lat2 long2 d2 3.1
E3 01:20:05 lat3 long3 d3 2.1
E4 08:15:04 lat4 long4 d4 3.5

To get the data in a shape that can act as input to a neural network (my first candidate is PredNet), it seems like there are two main options:

Time Windows

Say we decide on a 6 hour window starting at 00:00. Then E1, E2, E3 will fall in one window, E4 the next.  Which leads to the question of how to aggregate the first 3 events. Often events are geographically clustered, a large event will be associated with nearby foreshocks and aftershocks. For a first stab at this, it doesn’t seem unreasonable to assume such clustering will be the typical case. With this assumption, the data collapses down to :

[00:00-06:00] E2 lat2 long2 d2 3.1
[06:00-12:00] E4 lat4 long4 d4 3.5

This is lossy, so if say E1 and E2 were in totally different locations, the potentially useful information of E1 would be lost. A more sophisticated strategy would be to look for local clustering – not difficult in itself (check Euclidian distances), but then the question would be how to squeeze several event clusters into one time slot. As it stands it’s a simple strategy, and worth a try I reckon.

Time Differences

This strategy would involve a little transformation, like so:

E1[datetime]-E0[datetime] = ? lat1 long1 d1 2.2
E2[datetime]-E1[datetime] = 00:03:01 lat2 long2 d2 3.1
E3[datetime]-E2[datetime] = 00:02:01 lat3 long3 d3 2.1
E4[datetime]-E3[datetime] = 07:05:01 lat4 long4 d4 3.5

Now I must confess I really don’t know how much sense this makes, but it is capturing all the information, so it might just work. Again, it’s pretty simple and also worth a try.

I’d very much welcome comments and suggestions on this – do these strategies make sense? Are there any others that might be worth a try?

 

Seismic Data – fixed?

As described in my last post, I was seeing significant gaps in the seismic event data I was retrieving from the INGV service. So I re-read their docs. Silly me, I’d missed the option to include query arguments restricting the geo area of the events (I had code in a subsequent script doing this).

While tweaking the code to cover these parameters I also spotted a really clumsy mistake. I had a function doing more or less this –

for each event element in XML DOM:
        extract event data
        add event data to list
        return list

D’oh! Should have been –

for each event element in XML DOM:
        extract event data
        add event data to list
return list

I’ve also improved error handling considerably, discriminating between genuine HTTP errors and HTTP 204 No Content. Now I’ve narrowed the geo area and reduced the time window for each GET down to 1 hour, there are quite a lot of 204s.

I’m now running it over the time period around the l’Aquila quakes as a sanity check. Jeez, 20+ events in some hours, 10+ in most.

Assuming this works ok, I’ll run it over the whole 1997-2017 preiod, hopefully in ~12 hours time I’ll have some usable data.

PS. Looking good, for the 30 days following that of the l’Aquila big one, it produced:

in_zone_count = 8877
max_depth = 62800.0
max_magnitude = 6.1

 

 

 

Seismic Data Wrangling

Following my interim plan of proceeding software-only (until I’ve the funds to get back to playing with hardware), I’ve been looking at getting seismic event data from the INGV Web Service into a Keras/Tensorflow implementation of PredNet.

My code is on GitHub, and rather than linking to individual files which I may rename, I’ll put a README over there with pointers.

As a first step, I put together code to pull of the data and dump it down to simple CSV files. This appeared to be working. The demo implementation of PredNet takes HDF5 data from the KITTI  vision dataset (videos from a car on road around Karlsruhe), extracting it into numpy arrays, with the PredNet engine using Keras. To keep things simple I wanted to follow the same approach. I’m totally new to HDF5 so pinged Bill Lotter of the PredNet project for clues. He kindly gave me some helpful tips, and concurred with what I’d been thinking – keep the CSV data, process that into something PredNet can consume.

The data offered by the Web Service is good XML delivered over HTTP (props to INGV). But it does include a lot of material (provenance, measurement accuracy etc) that isn’t needed here. So my service-to-CSV code parses out just the relevant parts, producing a line for each event:

datetime, latitude, longitude, depth, magnitude

e.g.

2007-01-02T05:28:38.870000, 43.612, 12.493, 7700, 1.7

I couldn’t find the info anywhere, but it appears that the INGV service records go back at least to somewhere in the early 1990’s, so I chose 1997-01-01T00:00:00 as a convenient start datetime, giving me 20 years of events.

For this to be a similar shape to what PredNet expects, I will aggregate events within a particular time period (actually I think taking the most significant event in that period). I reckon 6 hour periods should be about right. This also seemed a reasonable window for calling the service (not). I’ll filter down the events to just those within the region of interest (northern Italy, see earlier post)  and then scale the latitude & longitude to an easy integer range (probably [128, 128]). For a first pass I’ll ignore the depth field.

As it happens, I’m well on the way to having implemented this. But along the way I did a few sanity checks, eg. checking for maximum event magnitude in the region of interest, (I got 4.1), and it turned out I was missing some rather significant data points.  Here’s one I checked for:

The 2009 L’Aquila earthquake occurred in the region of Abruzzo, in central Italy. The main shock occurred at 03:32 CEST (01:32 UTC) on 6 April 2009, and was rated 5.8 or 5.9 on the Richter magnitude scale and 6.3 on the moment magnitude scale; its epicentre was near L’Aquila, the capital of Abruzzo, which together with surrounding villages suffered most damage.

Nope, it wasn’t in the CSV, but the Web Service knows all about it:

http://webservices.ingv.it/fdsnws/event/1/query?eventId=1895389

Doing a service call for that whole day:

http://webservices.ingv.it/fdsnws/event/1/query?starttime=2009-04-06T00:00:00&endtime=2009-04-06T23:59:59

–  yields 877 events – nightmare day!

I’d set the timeout on the HTTP calls to 2 seconds, but there is so much data associated with each event that this was woefully inadequate. Since upped to 5 mins.

Manually checking calls, I was also sometimes getting a HTTP Status Code of 413 Request Entity Too Large. This puzzled me mightily – still does actually. It says request entity, not requested (or response) entity, but the way it’s behaving is that the response requested is too large. Either way I reckon the spec (latest is RFC7231) is a little open to misinterpretation here. (What the heck – I’ve mailed the IEFT HTTP list about it – heh, well well, I’ve co-chaired something with the chair…).

Anyhow, I’ve also tweaked the code to make calls over just 1 hour windows, hopefully it’ll now get the stuff it was missing.

Hmm…I’ve got it running now and it’s giving errors throughout the year 2000, which should be trouble-free. I think I’ll have to have it make several passes/retries to ensure I get the maximum data available.

Drat! It’s giving me Entity Too Large with just 1 hour windows, e.g.

http://webservices.ingv.it/fdsnws/event/1/query?starttime=2000-12-13T01:00:00&endtime=2000-12-13T02:00:00

I need to fix this…