Greg
animals brewing
food and drink gardening
general health
history language
music multimedia
opinion photography
politics Stones Road house
technology
Greg's diary
recent entries
Translate this page
Select day in April 2019:
Su Mo Tu We Th Fr Sa
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30
Select month:
2018 May Jun Jul Aug
2018 Sep Oct Nov Dec
2019 Jan Feb Mar Apr
Today's diary entry
Diary index
About this diary
Greg's home page
Greg's photos
Network link stats
Greg's other links
Copyright information
    
Groogle

Sunday, 21 April 2019 Dereel
Top of page
next day
last day

Still more PV software investigation
Topic: technology, opinion Link here

Yesterday's network sniffing gave some insight into how the photovoltaic inverter communicates, but not enough. Spent some more time sniffing with ngrep and found:

T 192.109.197.246:80 -> 192.109.197.137:51800 [AP]
  127..event: /ems/sse/stream.data: { "Devices": [ { "Id": "127.0.0.1:502:1", "Pac": 1563, "SetPoint": 0, "
  Alarms": "", "Status": "On-grid", "SOC": 31, "VBat": 265, "PacGrid": 0, "PacBat": 47, "PacPV": 1516, "Fro
  mPV": false, "Codes": "" }, { "Id": "127.0.0.1:502:1", "W1": -60, "W2": 0, "W3": 0 } ] }....EE..event: /e
  ms/sse/stream.data: { "Date": "2019-04-21 11:43:01", "Phase": 1, "W": 5, "Pac": 1541, "PacDischarge": 107
  , "PacCharge": 468, "PacPVBatt": 1117, "EMS_VBAT": 265, "EMS_SOC": 31, "WatDigitalInput": 0, "PacRev": 0,
   "HWOutputs": 0 }....E4..event: /ems/sse/stream.data: { "Date": "2019-04-21 11:43:01", "Phase": 2, "W": 0
  , "Pac": 0, "PacDischarge": 0, "PacCharge": 0, "PacPVBatt": 0, "EMS_VBAT": 265, "EMS_SOC": 31, "WatDigita
  lInput": 0, "PacRev": 0, "HWOutputs": 0 }....E4..event: /ems/sse/stream.data: { "Date": "2019-04-21 11:43
  :01", "Phase": 3, "W": 0, "Pac": 0, "PacDischarge": 0, "PacCharge": 0, "PacPVBatt": 0, "EMS_VBAT": 265, "
  EMS_SOC": 31, "WatDigitalInput": 0, "PacRev": 0, "HWOutputs": 0 }....

Ah! That's the real data, not the locations I found yesterday. And this one does have time stamps that make sense. OK, take a look at http://inverter.lemis.com/ems/sse/stream.data:

=== grog@eureka (/dev/pts/29) ~ 1 -> fetch http://inverter.lemis.com/ems/sse/stream.data
fetch: http://inverter.lemis.com/ems/sse/stream.data: Not Found

Hmm. What's this stream stuff? Must be something that I don't understand. But there's an alternative, the dirty one that Daniel O'Connor suggested: sniff the data stream and collect what I'm looking for. Here I have clearly identified source and destination IP addresses, source port, and the beginning of the packet data.

So how do I do that? BPF, of course, and more specifically pcap. Is there a program that will do this for me? You'd think that ngrep would do it for me, but it wasn't obvious from the man page.

Further searching brought me to this page, a tutorial with source code. Played around with that for a bit. The first program, ldev (source in the text), compiled out of the box and ran:

=== root@eureka (/dev/pts/4) /home/grog/src/pcap 107 -> ./ldev
DEV: em0
NET: 192.109.197.128
MASK: 255.255.255.128

Next came testpcap1.c. Again no issues compiling it, but then...

=== root@eureka (/dev/pts/4) /home/grog/src/pcap 108 -> ./testpcap1
DEV: em0
pcap_open_live(): em0: BIOCSRTIMEOUT: Invalid argument

OK, what's that? The code is:

    /* open the device for sniffing.

       pcap_t *pcap_open_live(char *device,int snaplen, int prmisc,int to_ms,
       char *ebuf)

       snaplen - maximum size of packets to capture in bytes
       promisc - set card in promiscuous mode?
       to_ms   - time to wait for packets in miliseconds before read
       times out
       errbuf  - if something happens, place error string here
     */

    descr = pcap_open_live(dev,BUFSIZ,0,-1,errbuf);

    if(descr == NULL)
    {
        printf("pcap_open_live(): %s\n",errbuf);
        exit(1);
    }

Where do I go from there? The best guess from the code is that it doesn't like -1 as a timeout value. But do I really want to get involved in debugging? I just want a filter. Maybe I should learn about streaming after all.


Irrigation woes again
Topic: gardening Link here

Yvonne told me what I should have known today: time to clean the bore water filter. I had decided that it needed cleaning every week rather than every few months, probably as a result of the low water level due to the dry summer. But I had neglected it for a couple of weeks, and it was very much in need of it.

That wasn't the only issue. It's been weeks (months?) since I decided I need to look at circuit 4, which has only too clearly been leaking:


This should be Irrigated-grass.jpeg.  Is it missing?
Image title: Irrigated grass          Dimensions:          5184 x 3888, 11922 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Monday, 22 April 2019, thumbnails          All images taken on Monday, 22 April 2019, small
Diary entry for Monday, 22 April 2019 Complete exposure details

 

Somehow I'm just too lazy.


Solar electricity progress
Topic: Stones Road house, technology, opinion Link here

We've been planning the solar electricity for well over 4 years, and much of the delay was due to procrastination on my part, coupled with a complete inability to establish contact with Simon Reid of BREAZE. Chris Bahlo didn't have that luxury: she has no grid connection, so she had to have it installed for her new house (also four years ago).

Today she came along and we compared notes. She paid about the same amount of money as we did. For that she got 7 (count them, seven) solar panels and a lead-acid battery of unspecified dimensions. But since she's only generating 1.75 kW, and the average hours of sunshine here are round 3.5 hours, it's unlikely that the batteries would store more than 7.5 kWh. And of course her inverter is completely dumb.

That's quite a difference, though even at the time I thought that the price was excessive. Maybe it wasn't such a bad thing after all that I didn't go with BREAZE.


Monday, 22 April 2019 Dereel Images for 22 April 2019
Top of page
previous day

Cracking the inverter data
Topic: technology, Stones Road house Link here

Yesterday I investigated sniffing the traffic between the inverter and a web browser, in the assumption that it would be easier than recreating the requests needed for communicating directly with the inverter. But Jamie Fraser and Daniel O'Connor (on IRC) saw it differently. Jamie suggested using the developer tools (F12 on most browsers, it seems). I've seen them before, but never used them in earnest, especially not to analyse HTTP streams. OK, try it out:

 
This should be devtools-1.png.  Is it missing?
Image title: devtools 1
Complete exposure details
Dimensions: 572 x 471, 171 kB
Dimensions of original: 1188 x 979, 171 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Monday, 22 April 2019:
thumbnails    small images    diary entry


This should be devtools-1-detail.png.  Is it missing?
Image title: devtools 1 detail          Dimensions:          1081 x 250, 126 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Display small version of all images on this page
All images taken on Monday, 22 April 2019, thumbnails          All images taken on Monday, 22 April 2019, small
Diary entry for Monday, 22 April 2019 Complete exposure details

 

That's not immediately obvious, but none of the traffic I'm looking for appears there. Ah, says Daniel, works on Chrome and Safari. OK, find a Chrome and try again. Great improvement. Now I can see the data coming in, and as expected without any GET requests:

 
This should be devtools-2.png.  Is it missing?
Image title: devtools 2
Complete exposure details
Dimensions: 514 x 525, 225 kB
Dimensions of original: 922 x 941, 210 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Monday, 22 April 2019:
thumbnails    small images    diary entry


This should be devtools-2-detail.png.  Is it missing?
Image title: devtools 2 detail          Dimensions:          2114 x 128, 121 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Display small version of all images on this page
All images taken on Monday, 22 April 2019, thumbnails          All images taken on Monday, 22 April 2019, small
Diary entry for Monday, 22 April 2019 Complete exposure details

 

Next, click on the left hand (name) column in the network tab. What column? I don't have no steenking columns. But Daniel did:


This should be f2IFrBH.png.  Is it missing?
Image title: f2IFrBH          Dimensions:          863 x 313, 97 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Display small version of all images on this page
All images taken on Monday, 22 April 2019, thumbnails          All images taken on Monday, 22 April 2019, small
Diary entry for Monday, 22 April 2019 Complete exposure details

 
This should be f2IFrBH-detail.png.  Is it missing?
Image title: f2IFrBH detail          Dimensions:          2104 x 128, 78 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Display small version of all images on this page
All images taken on Monday, 22 April 2019, thumbnails          All images taken on Monday, 22 April 2019, small
Diary entry for Monday, 22 April 2019 Complete exposure details

 

Ah, the console is hiding it all. Remove that and we have:

 
This should be devtools-3.png.  Is it missing?
Image title: devtools 3
Complete exposure details
Dimensions: 514 x 525, 151 kB
Dimensions of original: 922 x 941, 144 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Monday, 22 April 2019:
thumbnails    small images    diary entry


This should be devtools-3-detail.png.  Is it missing?
Image title: devtools 3 detail          Dimensions:          1404 x 192, 80 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Display small version of all images on this page
All images taken on Monday, 22 April 2019, thumbnails          All images taken on Monday, 22 April 2019, small
Diary entry for Monday, 22 April 2019 Complete exposure details

 

That looks a lot better. In particular, the progress bar at the right shows that it's continuous. And sure enough, select it as a curl command, and we're away!

=== grog@eureka (/dev/pts/30) ~ 11 -> curl 'http://inverter.lemis.com/system/events/sse/stream'  -H 'Accept: text/event-stream' -H 'Referer: http://inverter.lemis.com/dashboard/main.html' -H 'Cookie: __utma=51177012.766199467.1486957619.1542435223.1545448630.4; __utmc=51177012; __utmz=51177012.1542145939.2.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); adblk=adblk_no' -H 'Connection: keep-alive' -H 'Cache-Control: no-cache' --compressed
event: /ems/sse/stream
data: { "Devices": [ { "Id": "127.0.0.1:502:1", "Pac": 3704, "SetPoint": 0, "Alarms": "", "Status": "On-grid", "SOC": 100, "VBat": 268, "PacGrid": 0, "PacBat": 0, "PacPV": 3704, "FromPV": false, "Codes": "" }, { "Id": "127.0.0.1:502:1", "W1": -2816, "W2": 0, "W3": 0 } ] }

event: /ems/sse/stream
data: { "Devices": [ { "Id": "127.0.0.1:502:1", "Pac": 3724, "SetPoint": 0, "Alarms": "", "Status": "On-grid", "SOC": 100, "VBat": 268, "PacGrid": 0, "PacBat": -2, "PacPV": 3726, "FromPV": false, "Codes": "" }, { "Id": "127.0.0.1:502:1", "W1": -2805, "W2": 0, "W3": 0 } ] }

Wonderful!

What next? Pipe the data into a database, of course. Daniel has been there before as well, and he has an application, written in python, which collects the data, puts it in a database and even generates graphics. Took a look at it and was discouraged: it's relatively long, and my Python-fu is not really up to it. There must be a simpler, if kludgier, way.


Autumn lily
Topic: gardening Link here

Walking the dogs today, found this flower:

 
This should be Autumn-lily.jpeg.  Is it missing?
Image title: Autumn lily
Display location on map
Complete exposure details
Dimensions: 600 x 450, 139 kB
Dimensions of original: 5184 x 3888, 4384 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Monday, 22 April 2019:
thumbnails    small images    diary entry

I'm sure I've seen this one before, but it's worth identifying: it's a time of year when not much is flowering. Maybe I should find a few autumn-flowering bulbs.


This page contains (roughly) yesterday's and today's entries. I have a horror of reverse chronological documents, so all my diary entries are chronological. This page normally contains the last two days, but if I fall behind it may contain more. You can find older entries in the archive. Note that I often update a diary entry a day or two after I write it.     Do you have a comment about something I have written? This is a diary, not a “blog”, and there is deliberately no provision for directly adding comments. But I welcome feedback and try to reply to all messages I receive. See the diary overview for more details. If you do send me a message relating to something I have written, please indicate whether you'd prefer me not to mention your name. Otherwise I'll assume that it's OK to do so.


Greg's home page This month Greg's photos Greg's links

RSS 2.0 Valid XHTML 1.0!