Capturing Dynamic Web Data

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
I want to capture my router's WAN IP address, but I cannot simply parse the HTML code as I thought, because the data is dynamically generated and sent to my browser. I've already tried this, and it does not work. My next bright idea is to try to capture it as network traffic. I'm thinking about using Wire Shark to see what the traffic looks like and then use a non-gui program to capture and store the data. Sound like a plan? Anyone know of an easy way to do this?
 

WBahn

Joined Mar 31, 2012
29,979
I don't know that the WAN address will be visible on the LAN side of the router. Depends on how things are configured. Can you talk to the router and ask it what it's IP address is? What router do you have and how are you using it (access point, bridge, etc.)?
 

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
The only way I know of to query my router is to request the Control and Status webpage, served locally, and only available to the LAN connected devices. This has been the case in every router I've owned. However, I'm not interested in looking at a webpage ( in the context of this thread ) rather, I want to capture the IP address for processing by a script. Last night, I was able to capture the IP address as Ethernet traffic using Wire Shark. But more importantly, I was able to drill down to the actual form that gets loaded and imbedded into the main webpage. Within the form is this code:

Code:
<input type="hidden" name="runtest" value="no">
<INPUT name=wan_proto type=hidden value= "dhcp">
<INPUT name=wan_ipaddr type=hidden value= "XX.XX.XX.XX">
<INPUT name=wan_netmask type=hidden value= "255.255.254.0">
<INPUT name=wan_gateway type=hidden value= "XX.XX.XX.XX">
<INPUT name=wan_dns_sel type=hidden value= "0">
<INPUT name=wan_dns1_pri type=hidden value= "0.0.0.0">
<INPUT name=wan_dns1_sec type=hidden value= "">
The X's are mine. Now all that's left is to parse the text and extract the IP address. I came across "cURL" as a means of streaming/piping the HTML to a script for parsing, something like this:

curl "http://10.0.0.1/BAS_ether.htm" | my_parse_script

The script should be a simple awk listing. I haven't tried this yet. I'll try it when I finish my 'homeydo' list today.
 

joeyd999

Joined Jun 6, 2011
5,237
Last night, I was able to capture the IP address as Ethernet traffic using Wire Shark.
Why do you believe local traffic has embedded within it your external WAN address? Please, tell me the RFC # that details this.

I am sure Wire Shark is pulling data in from outside your WAN, probably from one of its own servers.

But more importantly, I was able to drill down to the actual form that gets loaded and imbedded into the main webpage. Within the form is this code:

Code:
<input type="hidden" name="runtest" value="no">
<INPUT name=wan_proto type=hidden value= "dhcp">
<INPUT name=wan_ipaddr type=hidden value= "XX.XX.XX.XX">
<INPUT name=wan_netmask type=hidden value= "255.255.254.0">
<INPUT name=wan_gateway type=hidden value= "XX.XX.XX.XX">
<INPUT name=wan_dns_sel type=hidden value= "0">
<INPUT name=wan_dns1_pri type=hidden value= "0.0.0.0">
<INPUT name=wan_dns1_sec type=hidden value= "">
What is the source of the form HTML? I'd be interested in seeing the whole <FORM> tag.
 

sirch2

Joined Jan 21, 2013
1,037
I assume you want to do this because your ISP changes yourexternal IP address regularly? Have you considered dynamic dns services such as NoIP?
 

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
Why do you believe local traffic has embedded within it your external WAN address.
I said the WAN address is provided in the control and status webpage, served locally.

Please, tell me the RFC # that details this
Who said anything about an RFC#??? Look up whatever RFC# covers serving a webpage

I am sure Wire Shark is pulling data in from outside your WAN, probably from one of its own servers.
Nope. The webpage is served locally from the router. That's how every router I've owned works. Wire Shark monitors traffic on the local device. It does not get remote data.



What is the source of the form HTML? I'd be interested in seeing the whole <FORM> tag.
Not sure what you expect to see, but here goes.

<formname="formname"method="POST"action="ether.cgi"target="_parent">

<tableborder="0"cellpadding="0"cellspacing="3"width="100%">
<tr>
<tdnowrap><ahref="javascript:loadhelp('BAS_ether','accountname')"tabindex="-1"><b>Account Name</b>&nbsp;(If Required)</a></td>
<tdalign="right"><inputtype="text"name="system_name"maxlength="60"size="15"value="WNR2000v2"></td>
</tr>
<tr>
<tdnowrap><ahref="javascript:loadhelp('BAS_ether','Domain')"tabindex="-1"><b>Domain Name</b>&nbsp;(If Required)</a></td>
<tdalign="right"><inputtype="text"name="domain_name"maxlength="60"size="15"value=""></td>
</tr>
<tr><!-- RULE -->
<tdcolspan="2"background="liteblue.gif"height="12">&nbsp;</td>
</tr>
<tr>
<tdcolspan="2"nowrap><ahref="javascript:loadhelp('BAS_ether','InternetIP')"tabindex="-1"><b>Internet IP Address</b></a></td>
</tr>
<tr>
<tdcolspan="2"nowrap><inputtype="radio"name="WANAssign"value="dhcp"onClick="setIP()">Get Dynamically from ISP</td>
</tr>
<tr>
<tdcolspan="2"nowrap><inputtype="radio"name="WANAssign"value="static"onClick="setIP()">Use Static IP Address</td>
</tr>
<tr>
<tdnowrap><imgsrc="spacer.gif"width="20"height="12"border="0">IP Address</td>
<tdnowrapalign="right"class="num"><inputtype="text"name="WPethr1"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()">.<inputtype="text"name="WPethr2"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()">.<inputtype="text"name="WPethr3"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()">.<inputtype="text"name="WPethr4"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()"></td>
</tr>
<tr>
<tdnowrap><imgsrc="spacer.gif"width="20"height="12"border="0">IP Subnet Mask</td>
<tdalign="right"class="num"><inputtype="text"name="WMask1"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()">.<inputtype="text"name="WMask2"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()">.<inputtype="text"name="WMask3"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()">.<inputtype="text"name="WMask4"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()"></td>
</tr>
<tr>
<tdnowrap><imgsrc="spacer.gif"width="20"height="12"border="0">Gateway IP Address</td>
<tdalign="right"class="num"><inputtype="text"name="WGateway1"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()">.<inputtype="text"name="WGateway2"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()">.<inputtype="text"name="WGateway3"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()">.<inputtype="text"name="WGateway4"class="num"size="4"maxlength="3"onFocus="if(DisableFixedIP) this.blur()"></td>
</tr>
<tr><!-- RULE -->
<tdcolspan="2"background="liteblue.gif"height="12">&nbsp;</td>
</tr>
<tr>
<tdnowrapcolspan="2"><ahref="javascript:loadhelp('BAS_ether','DNSaddress')"tabindex="-1"><b>Domain Name Server (DNS) Address</b></a></td>
</tr>
<tr>
<tdcolspan="2"nowrap><inputtype="radio"name="DNSAssign"value="0"onClick="setDNS()">Get Automatically from ISP</td>
</tr>
<tr>
<tdcolspan="2"nowrap><inputtype="radio"name="DNSAssign"value="1"onClick="setDNS()">Use These DNS Servers</td>
</tr>
<tr>
<tdnowrap><imgsrc="spacer.gif"width="20"height="12"border="0">Primary DNS</td>
<tdnowrapalign="right"class="num"><inputtype="text"name="DAddr1"class="num"size="4"maxlength="3"onFocus="if(DisableFixedDNS) this.blur()">.<inputtype="text"name="DAddr2"class="num"size="4"maxlength="3"onFocus="if(DisableFixedDNS) this.blur()">.<inputtype="text"name="DAddr3"class="num"size="4"maxlength="3"onFocus="if(DisableFixedDNS) this.blur()">.<inputtype="text"name="DAddr4"class="num"size="4"maxlength="3"onFocus="if(DisableFixedDNS) this.blur()"></td>
</tr>
<tr>
<tdnowrap><imgsrc="spacer.gif"width="20"height="12"border="0">Secondary DNS</td>
<tdnowrapalign="right"class="num"><inputtype="text"name="PDAddr1"class="num"size="4"maxlength="3"onFocus="if(DisableFixedDNS) this.blur()">.<inputtype="text"name="PDAddr2"class="num"size="4"maxlength="3"onFocus="if(DisableFixedDNS) this.blur()">.<inputtype="text"name="PDAddr3"class="num"size="4"maxlength="3"onFocus="if(DisableFixedDNS) this.blur()">.<inputtype="text"name="PDAddr4"class="num"size="4"maxlength="3"onFocus="if(DisableFixedDNS) this.blur()"></td>
</tr>
<tr><!-- RULE -->
<tdcolspan="2"background="liteblue.gif"height="12">&nbsp;</td>
</tr>
<tr>
<tdnowrapcolspan="2"><ahref="javascript:loadhelp('BAS_ether','localaddress')"tabindex="-1"><b>Router MAC Address</b></a></td>
</tr>
<tr>
<tdnowrapcolspan="2"><inputtype="radio"name="MACAssign"value="0"onClick="setMAC()">Use Default Address</td>
</tr>
<tr>
<tdnowrapcolspan="2"><inputtype="radio"name="MACAssign"value="1"onClick="setMAC()">Use Computer MAC Address</td>
</tr>
<tr>
<tdnowrapalign="left"><inputtype="radio"name="MACAssign"value="2"onClick="setMAC()">Use This MAC Address</td>
<tdnowrapalign="right"class="num"><inputtype="text"class="num"name="Spoofmac"size="17"maxlength="17"value=""onFocus="if(DisableFixedMAC) this.blur()">
</td>
</tr>

<tr><!-- RULE -->
<tdcolspan="2"background="liteblue.gif"height="12">&nbsp;</td>
</tr>

<tr>
<tdcolspan="2"align="center"><divID="basbuttons"onmouseover="loadhelp('BAS_ether','buttons')">
<inputtype="SUBMIT"name="apply"value=ApplyonClick="return checkData()">
<inputtype="BUTTON"name="Cancel"value=CancelonClick="reset();loadSettings();loadcheck();">
<inputtype="SUBMIT"name="Test"value=TestonClick="return checkData()">
</div></td></tr>
</table>
<inputtype="hidden"name="runtest"value="no">
<INPUTname=wan_prototype=hiddenvalue="dhcp">
<INPUTname=wan_ipaddrtype=hiddenvalue="75.76.21.34">
<INPUTname=wan_netmasktype=hiddenvalue="255.255.254.0">
<INPUTname=wan_gatewaytype=hiddenvalue="75.76.20.1">
<INPUTname=wan_dns_seltype=hiddenvalue="0">
<INPUTname=wan_dns1_pritype=hiddenvalue="0.0.0.0">
<INPUTname=wan_dns1_sectype=hiddenvalue="">
<INPUTname=wan_hwaddr_seltype=hiddenvalue="2">
<INPUTname=wan_hwaddr_deftype=hiddenvalue="00:13:20:30:A6:70">
<INPUTname=wan_hwaddr2type=hiddenvalue="00:13:20:30:A6:70">
<INPUTname=wan_hwaddr_pctype=hiddenvalue="00:21:6A:BC:C8:5A">
<INPUTname=lan_ipaddrtype=hiddenvalue="10.0.0.1">
<INPUTname=lan_netmasktype=hiddenvalue="255.255.255.0">
</form>
 
Last edited:

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
I assume you want to do this because your ISP changes yourexternal IP address regularly? Have you considered dynamic dns services such as NoIP?
You are correct. Yes I do play to use DDNS. This effort is part of an overall strategy to implement remote debugging in case of a system failure. I want to use a completely separate communication system (eg 4G) to view system information if I experience an internet failure. This will enable me to better know what corrective action to take.
 

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
All the pertinents are given at the very bottom, in the <input> statements. All I need to do is parse these statements. At first I thought I would need to monitor traffic and extract it. That was because, when I view the webpage, this form gets embedded by some magic and I could not see the form data when I looked at the source. But now I realize I can simple request the form itself by the method I described above. So all is well now.
 

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
Read about web RTC.
Thanks, I think it would be helpful if I could both read information about my network using the locally served webpage, and also use a program to mine information from the web. That would allow me to create a 'tiered' diagnostic scheme. If the first tier is successful, then I could proceed to requesting information from the www. That would be a more detailed diagnostic. Right now I have this on hold, since I think I have a good solution, while I am working on another aspect of this project ( posting about it in a moment, because I've hit another snag)
 

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
Update: I was thinking about this effort tonight and decided to look up the thread. I did get on DDNS and it's working fine. I can log on to my home server and see and control my security cams. However, I don't have any way to evaluate the system from a remote location. For example, I am working in Minneapolis right now, and when I have an outtage, I lose my server and have to way to ascertain what the problem is. Fortunately, I can call my wife and find out, but she will be joining me soon, and so there will be nobody there to help. So, I plan on having some 'backup' system to communicate with the local network and read the IP address by the method described above. I plan on using DTMF or something simple that doesn't require a working internet.
 
Top