Capturing Dynamic Web Data

Discussion in 'Computing and Networks' started by Brownout, Feb 8, 2015.

  1. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    I want to capture my router's WAN IP address, but I cannot simply parse the HTML code as I thought, because the data is dynamically generated and sent to my browser. I've already tried this, and it does not work. My next bright idea is to try to capture it as network traffic. I'm thinking about using Wire Shark to see what the traffic looks like and then use a non-gui program to capture and store the data. Sound like a plan? Anyone know of an easy way to do this?
     
  2. WBahn

    Moderator

    Mar 31, 2012
    17,720
    4,788
    I don't know that the WAN address will be visible on the LAN side of the router. Depends on how things are configured. Can you talk to the router and ask it what it's IP address is? What router do you have and how are you using it (access point, bridge, etc.)?
     
  3. joeyd999

    AAC Fanatic!

    Jun 6, 2011
    2,675
    2,724
    Like WBahn said, the local network has no concept of the WAN. You'll either need to query your router, or query an outside IP service like:

    http://wanip.info/
     
  4. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    The only way I know of to query my router is to request the Control and Status webpage, served locally, and only available to the LAN connected devices. This has been the case in every router I've owned. However, I'm not interested in looking at a webpage ( in the context of this thread ) rather, I want to capture the IP address for processing by a script. Last night, I was able to capture the IP address as Ethernet traffic using Wire Shark. But more importantly, I was able to drill down to the actual form that gets loaded and imbedded into the main webpage. Within the form is this code:

    Code (Text):
    1. <input type="hidden" name="runtest" value="no">
    2. <INPUT name=wan_proto type=hidden value= "dhcp">
    3. <INPUT name=wan_ipaddr type=hidden value= "XX.XX.XX.XX">
    4. <INPUT name=wan_netmask type=hidden value= "255.255.254.0">
    5. <INPUT name=wan_gateway type=hidden value= "XX.XX.XX.XX">
    6. <INPUT name=wan_dns_sel type=hidden value= "0">
    7. <INPUT name=wan_dns1_pri type=hidden value= "0.0.0.0">
    8. <INPUT name=wan_dns1_sec type=hidden value= "">
    9.  
    The X's are mine. Now all that's left is to parse the text and extract the IP address. I came across "cURL" as a means of streaming/piping the HTML to a script for parsing, something like this:

    curl "http://10.0.0.1/BAS_ether.htm" | my_parse_script

    The script should be a simple awk listing. I haven't tried this yet. I'll try it when I finish my 'homeydo' list today.
     
  5. joeyd999

    AAC Fanatic!

    Jun 6, 2011
    2,675
    2,724
    Why do you believe local traffic has embedded within it your external WAN address? Please, tell me the RFC # that details this.

    I am sure Wire Shark is pulling data in from outside your WAN, probably from one of its own servers.

    What is the source of the form HTML? I'd be interested in seeing the whole <FORM> tag.
     
  6. sirch2

    Well-Known Member

    Jan 21, 2013
    1,008
    351
    I assume you want to do this because your ISP changes yourexternal IP address regularly? Have you considered dynamic dns services such as NoIP?
     
  7. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    I said the WAN address is provided in the control and status webpage, served locally.

    Who said anything about an RFC#??? Look up whatever RFC# covers serving a webpage

    Nope. The webpage is served locally from the router. That's how every router I've owned works. Wire Shark monitors traffic on the local device. It does not get remote data.



    Not sure what you expect to see, but here goes.

     
    Last edited: Feb 11, 2015
  8. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    You are correct. Yes I do play to use DDNS. This effort is part of an overall strategy to implement remote debugging in case of a system failure. I want to use a completely separate communication system (eg 4G) to view system information if I experience an internet failure. This will enable me to better know what corrective action to take.
     
  9. joeyd999

    AAC Fanatic!

    Jun 6, 2011
    2,675
    2,724
    Sorry, I misunderstood what you were doing with wireshark.

    Didn't see what I was looking for. Sorry...can't help.
     
  10. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    All the pertinents are given at the very bottom, in the <input> statements. All I need to do is parse these statements. At first I thought I would need to monitor traffic and extract it. That was because, when I view the webpage, this form gets embedded by some magic and I could not see the form data when I looked at the source. But now I realize I can simple request the form itself by the method I described above. So all is well now.
     
  11. BR-549

    Well-Known Member

    Sep 22, 2013
    1,981
    388
  12. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    Thanks, I think it would be helpful if I could both read information about my network using the locally served webpage, and also use a program to mine information from the web. That would allow me to create a 'tiered' diagnostic scheme. If the first tier is successful, then I could proceed to requesting information from the www. That would be a more detailed diagnostic. Right now I have this on hold, since I think I have a good solution, while I am working on another aspect of this project ( posting about it in a moment, because I've hit another snag)
     
  13. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    Update: I was thinking about this effort tonight and decided to look up the thread. I did get on DDNS and it's working fine. I can log on to my home server and see and control my security cams. However, I don't have any way to evaluate the system from a remote location. For example, I am working in Minneapolis right now, and when I have an outtage, I lose my server and have to way to ascertain what the problem is. Fortunately, I can call my wife and find out, but she will be joining me soon, and so there will be nobody there to help. So, I plan on having some 'backup' system to communicate with the local network and read the IP address by the method described above. I plan on using DTMF or something simple that doesn't require a working internet.
     
Loading...