PF_PACKET Protocol Family

source

When you open a socket with the standard call sock = socket(domain, type, protocol) you have to specify which domain (or protocol family) you are going to use with that socket. Commonly used families are PF_UNIX, for communications bounded on the local machine, and PF_INET, for communications based on IPv4 protocols. Furthermore, you have to specify a type for your socket and possible values depend on the family you specified. Common values for type, when dealing with the PF_INET family, include SOCK_STREAM (typically associated with TCP) and SOCK_DGRAM (associated with UDP). Socket types influence how packets are handled by the kernel before being passed up to the application. Finally, you specify the protocol that will handle the packets flowing through the socket (more details on this can be found on the socket(3) man page).

In recent versions [original article was written in 2001] of the Linux kernel (post-2.0 releases) a new protocol family has been introduced, named PF_PACKET. This family allows an application to send and receive packets dealing directly with the network card driver, thus avoiding the usual protocol stack-handling (e.g., IP/TCP or IP/UDP processing). That is, any packet sent through the socket will be directly passed to the Ethernet interface, and any packet received through the interface will be directly passed to the application.

The PF_PACKET family supports two slightly different socket types, SOCK_DGRAM and SOCK_RAW. The former leaves to the kernel the burden of adding and removing Ethernet level headers. The latter gives the application complete control over the Ethernet header. The protocol field in the socket() call must match one of the Ethernet IDs defined in /usr/include/linux/if_ether.h [IEEE 802.3 Ethernet magic constants], which represents the registered protocols that can be shipped in an Ethernet frame. Unless dealing with very specific protocols, you typically use ETH_P_IP, which encompasses all of the IP-suite protocols (e.g., TCP, UDP, ICMP, raw IP and so on).

Since they have pretty serious security implications (for example, you may forge a frame with a spoofed MAC address), PF_PACKET-family sockets may only be used by root.

Ruby Raw Socket for Windows

About Raw Sockets in general:
- Wikipedia article: A raw socket is a socket that allows direct sending and receiving of network packets by applications, bypassing all encapsulation in the networking software of the operating system.

Linux specific:
- terminal type:

man 7 raw

to read about raw sockets
- might be useful #1
- might be useful #2


Windows specific:
- libpcap-1.1.1./pcap-win32.c has the wsockinit() function, not very useful though
- Windows uses Winsock(Wiki article)
- Very good Windows raw socket explaination also with code: Raw Sockets and Windows
First of all, it must be understood very clearly that raw sockets is not a feature of the network API (although it must be present there as an option) but of the OS protocol stack. To implement raw sockets, all we have to do is to inform the OS that the packet buffer we are providing will have the header and so the OS should transmit it as is without "adding any header"; that's all, nothing more to do. The Unix operating system has raw socket support since ancient times. But the problem is with Windows. None of Windows 95, 98, 98SE supported raw sockets. Raw sockets became available on Windows from Windows 2000; Windows XP continued this. But suddenly, raw socket support was removed from Windows XP through a patch in SP2. Vista probably doesn't have it. Windows 95, 98, 98SE do not support raw sockets, but this doesn't end the story. If you want the facility, then the solution is to use a third party packet driver like Winpcap. Such packet drivers will do your task irrespective of what the OS likes and dislikes. Windows XP and XP SP1 have full raw socket support and so life is easy. So if you want to do raw socketing on Windows, then either use Winpcap or don't feel desperate to install SP2, or otherwise use Windows 2003 which, as per my knowledge, has raw socket support. So let's brief up.
-----
Windows 95, 98, 98SE, NT4.0 -- Only raw ICMP and IGMP with restricted features.
Windows 2000, XP, XP SP1, 2003 -- Full raw socket support for both receiving and sending purposes.
Windows XP SP2 -- Only raw ICMP, IGMP, and UDP with proper source address (IP spoofing restricted) can be sent. But, full raw sockets can be received, which means you can sniff all incoming data and read their headers.
Note : Winsock Ver. >=2.0
----
So if your system doesn't support raw sockets, then switch to Linux or use Winpcap.

- Very useful Winsock FAQ
--- question: Do I must have a TCP/IP?
) What do I need to run WinSock applications?
----------------------------------------------
Interesting how after "rake install" in Windows, the Gemfile.lock shows "x86-mingw32" as well as "ruby" under "PLATFORMS"

----------

Using WinSock applications to access the Internet requires:

- A suitable connection to the Internet.
- A TCP/IP stack (which includes it's own WINSOCK.DLL).

=====================
Trying to see if existing Linux socket would work on Windows:

commit 9a5b019bea6f62f22cf90e2c6eaf0c0637387e7f
take out unnecessary raise exception. Hopefully windows socket can work?

Administrator@HAOQI-4F34C3203 ~/Desktop/ethernet/spec/ethernet (master)
$ rspec raw_socket_spec.rb
FFFFFF

Failures:

  1) RawSocket mac should have 6 bytes
     Failure/Error: let(:mac) { Ethernet::RawSocket.mac eth_device }
     RuntimeError:
       Unsupported platform i386-mingw32
     # c:/Documents and Settings/Administrator/Desktop/ethernet/lib/ethernet/raw
_socket.rb:121:in `all_ethernet_protocols'
     # c:/Documents and Settings/Administrator/Desktop/ethernet/lib/ethernet/raw
_socket.rb:24:in `socket'
     # c:/Documents and Settings/Administrator/Desktop/ethernet/lib/ethernet/raw
_socket.rb:49:in `mac'
     # ./raw_socket_spec.rb:8:in `block (2 levels) in '
     # ./raw_socket_spec.rb:17:in `block (3 levels) in '

  2) RawSocket mac should match ifconfig output
     same as above

  3) RawSocket socket should be able to receive data
     Failure/Error: before { @socket = Ethernet::RawSocket.socket eth_device }
     RuntimeError:
       Unsupported platform i386-mingw32
     # c:/Documents and Settings/Administrator/Desktop/ethernet/lib/ethernet/raw
_socket.rb:121:in `all_ethernet_protocols'
     # c:/Documents and Settings/Administrator/Desktop/ethernet/lib/ethernet/raw
_socket.rb:24:in `socket'
     # ./raw_socket_spec.rb:28:in `block (3 levels) in '

  4) RawSocket socket should output a packet
     same as above

  5) RawSocket socket should receive some network noise
     same as above

  6) RawSocket testing listing all network devices should list the devices
     Failure/Error: device_macaddrs.each do |key, value|
     NoMethodError:
       undefined method `each' for nil:NilClass
     # ./raw_socket_spec.rb:48:in `block (3 levels) in '

Finished in 0 seconds
6 examples, 6 failures

Currently, I have all the errors because of the highlighted line above. Looking at that:

23   def self.socket(eth_device = nil, ether_type = nil)
 24     ether_type ||= all_ethernet_protocols
 25     socket = Socket.new raw_address_family, Socket::SOCK_RAW, htons(ether_type)
...
115     # The protocol number for listening to all ethernet protocols.
116     def all_ethernet_protocols
117       case RUBY_PLATFORM
118       when /linux/
119         3
120       else
121         raise "Unsupported platform #{RUBY_PLATFORM}"
122       end
123     end

What "||=" means in "ether_type ||= all_ethernet_protocols"

Victor's answer:
It's a trick in modern languages. || is boolean OR, and the way it's implemented is a || b returns a if it is a true value, otherwise it returns b. You can think of it for a bit and convince yourself that this fulfills the contract for OR. ||= is used to specify default values -- if ether_type was nil or false, it becomes all_ethernet_protocols. Otherwise it doesn't change.

1. so ether_type, if it's false or nil, is going to be replaced by "3", how can that be compatible with the 4-digit ether_type?
2. is 3 the Gateway-to-Gateway protocol?
answer: The number comes from cat /usr/include/linux/if_ether.h | grep ETH_P_ALL, which along with PF_Packet, make up a socket. 

Also for this:

140     # The AF / PF number for raw sockets.
141     def raw_address_family
142       case RUBY_PLATFORM
143       when /linux/
144         17 # cat /usr/include/bits/socket.h | grep PF_PACKET
145       when /darwin/
146         18 # cat /usr/include/sys/socket.h | grep AF_LINK

147       when /i386-mingw32/ 
148         18 # winsock.h | grep AF_LINK
149       else
150         raise "Unsupported platform #{RUBY_PLATFORM}"
151       end
152     end

Sweet! Found the Windows equivalent of socket.h -- winsock.h!!  I hope it's up to date. So it says that AF_LINK is 18. :D
Victor says: "I googled around, and it seems that AF_LINK won't work on Windows. The header you have is for Winsock 1.1, and all modern systems use Winsock 2.0. MSDN says that Windows removed raw socket support as of Windows XP SP2." This was done for security reasons.

Window's (Winsock/WSA/Windows Socket API) Raw Socket. Raw Socket is a kind of Windows socket (also containing some common choices for domain, type, and protocol), following the format of Berkeley's raw socket.
domain: AF_LINK, same as /darwin/
type: SOCK_RAW, same :D
protocol: hope that 3, eth_p_all would work
:P

commit b458ffaebeabc408c9649c37b9e1951316852ab5
try still. Fixed some Windows socket stuff, found AF_LINK in winsock.h

Administrator@HAOQI-4F34C3203 ~/Desktop/ethernet/spec/ethernet (master)
$ rspec raw_socket_spec.rb
FFFFFF

Failures:

  1) RawSocket mac should have 6 bytes
     Failure/Error: let(:mac) { Ethernet::RawSocket.mac eth_device }
     Errno::EAFNOSUPPORT:
       An address incompatible with the requested protocol was used. - socket(2)

     # c:/Documents and Settings/Administrator/Desktop/ethernet/lib/ethernet/raw
_socket.rb:25:in `initialize'
     # c:/Documents and Settings/Administrator/Desktop/ethernet/lib/ethernet/raw
_socket.rb:25:in `new'
     # c:/Documents and Settings/Administrator/Desktop/ethernet/lib/ethernet/raw
_socket.rb:25:in `socket'
     # c:/Documents and Settings/Administrator/Desktop/ethernet/lib/ethernet/raw
_socket.rb:49:in `mac'
     # ./raw_socket_spec.rb:8:in `block (2 levels) in '
     # ./raw_socket_spec.rb:17:in `block (3 levels) in '

... same stuff

Finished in 0.01562 seconds
6 examples, 6 failures

Failure/Error: let(:mac) { Ethernet::RawSocket.mac eth_device }
     NameError:
       uninitialized constant Module::ANY
     # c:/Documents and Settings/Administrator/Desktop/ethernet/lib/ethernet/raw
_socket.rb:133:in `all_ethernet_protocols'

./Win32/Src/getaddrinfo.c: { PF_INET, SOCK_RAW, ANY, NULL, 0x05 },

Try instead of htons(ETH_P_ALL) , which we have here as 3, do htons(ANY)

commit 6be0d35378b021a8981dfbec4b920317123bc64d

try. switch htons(3) to htons(ANY)

question for Victor:
Do you have any hints of looking for the Windows equivalent of if_ether.h or know what I should put in for ether_type?

Continued from Victor's quote above ... "All solutions seem to revolve around installing some NDIS driver and talking to it. Winpcap seems to be the easiest and most well-maintained library for that."

I need raw socket for an application that i am trying to execute from cygwin in 'Windows XP with SP2' & Vista . How can this be achieved ?
Use a driver that doesn't use the Windows TCP/IP stack, maybe try WinPcap.



====

- Download WinPcap source
-> install NDIS driver
Nice Site

Making the RubyGem work (bundle & rake install) with Windows - ruby for different platforms!

Ruby file .rb:
Detecting operating systems on Ruby, the backslashes are for regex:
    Linux: /linux/
    Windows: /win32/
    Mac OS: /darwin/
So a ruby file would have these things to distinguish different code for different platforms:

case RUBY_PLATFORM
  when /i386-mingw32/ # OR /win32/ matching Gemfile.lock under PLATFORMS
    require_files :win32
  when /linux/
    require_files :linux

else
    raise "Unsupported platform #{RUBY_PLATFORM}"
end

Gemfile:
Gem Bundler's Gemfile has special syntax (documentation, and examples).

Here is an example to write Windows-only gem:

### windows specific gems ###
 gem 'win32-process', :platforms => [:mswin, :mingw]
 gem 'win32console', :platforms => [:mswin, :mingw]

or 

### windows specific gems ###
platforms :mswin, :mingw do
    gem  'win32console', '1.3.0'
    gem  'wind32-process'
end

The symbols ":mswin" and ":mingw" are special platforms groups.

gem "rspec", :group => :test
gem "wirble", :groups => [:development, :test]

Unfortunately, there is no ":linux", so it'll be a non-platform group, which means the command should be "bundle install --without linux" when running on Windows.

Yay! Can:

bundle install --without linux
rake install

commit 8b8573dcc87e673cca042e2d68569fffd586e872
try if Windows rake will work after making system/getifaddrs only for linux


====
Next step:
- downgrade to 1.8.7 ruby
- make sockets work  (google, more wireshark digging)
- How to list devices, look at: fad-win32.c.
(change :linux to :ruby to avoid "--without linux" on Windows, i.e. does :ruby work as platform thingy not just a group?)
===
IRC: freenode ruby chat
#RubyOnRails
#bundler
#ruby-lang

==
This pipe server Windomit ws gem be helpful?

todo

0. Can I incorporate system-getifaddr?

• Yes, and add "system/getifaddrs" to the list of required gems. No need for requiring rubygems

1. What's wrong with ethernet_ping? Solved
2. Why can't I rspec? Fixed
3. list devices (all os)
- add in the mac address listing thing
- fix the require system/getifaddrs
- make it possible to rake install on windows!! (Gemfile has special syntax, rest of "system-ifaddrs" just regex).
- downgrade windows ruby from 1.9.2 to 1.8.7


- add enable_pcap, ether_ping_server

Listing all network interfaces/devices and their MAC Addresses (for Linux with Ruby)

I used this gem to get System.get_ifaddrs() working on Ruby.

(Testing in ether_shell because my ethernet still can't pass rspec.)

For the 'system/getifaddrs' require, have to let user install it before they can use it, so do:
1. Add into the Gemfile, like

  2 # Add dependencies required to use your gem here.
  3 # Example:
  4 #   gem "activesupport", ">= 2.3.5"
  5   gem "system-getifaddrs", "~> 0.1.1"

2. Do NOT add gem.add_runtime_dependency 'rcov', '>= 0.1.1' to Rakefile because it's automatic now
3. bundle
4. sudo rake install

Add in lib/ethernet/raw_socket.rb:

require 'system/getifaddrs' # for listing

...

  # returns a symbols array of all network devices
  def self.list_all_devices()
    case RUBY_PLATFORM
    when /linux/ #TODO: test to see if works on MacOS as well 
      devicehash = System.get_ifaddrs # {:eth0=>{:inet_addr=>"18.248.7.89", :netmask=>"18.248.7.89"}, :vmnet1=>{:inet_addr=>"172.16.76.1", :netmask=>"172.16.76.1"}}
      devicehash.keys
    end 
  end 

  # returns a hash of all network devices => MAC Address
  def self.list_all_devices_macaddr()
    case RUBY_PLATFORM
    when /linux/ 
      devices = self.list_all_devices
      device_macaddrs = {}
   
      devices.each do |eth_device|
        device_macaddrs[eth_device] = self.mac(eth_device.to_s).unpack('H*')[0]
      end 
      device_macaddrs
    end 
  end 

Add in spec/ethernet/raw_socket_spec.rb

describe 'testing listing all network devices' do
     it 'should list the devices' do
       device_macaddrs = Ethernet::RawSocket.list_all_devices_macaddr()
       device_macaddrs.each do |key, value|
         puts "\nDevice is: "
         puts key.to_s
         puts "MAC Addrs: "
         puts value
         puts "\n"
       end 
     end 
  end 

Results:

$ sudo rspec raw_socket_spec.rb 
.....
Device is: 
vmnet1
MAC Addrs: 
005056c00001


Device is: 
vmnet8
MAC Addrs: 
005056c00008


Device is: 
lo
MAC Addrs: 
000000000000


Device is: 
eth0
MAC Addrs: 
002170925bc5

.

Finished in 0.11353 seconds
6 examples, 0 failures

Details of how I found this solution, very boring

I could run ether_shell's specs. Specifically, sudo rspec raw_socket_spec.rb, which currently has eth0 as the device

3 describe 'RawSocket' do
  4   let(:eth_device) { 'eth0' } 

Now I want to see if the outputting mac address function works.

In raw_socket.rb, .mac() is the function that gets the mac address of a device:

28   # The MAC address for an Ethernet card.
 29   #
 30   # Args:
 31   # eth_device:: device name for the Ethernet card, e.g. 'eth0'
 32   def self.mac(eth_device)
 33     case RUBY_PLATFORM 
 34     when /linux/
 35       # /usr/include/net/if.h, structure ifreq
 36       ifreq = [eth_device].pack 'a32'
 37       # 0x8927 is SIOCGIFHWADDR in /usr/include/bits/ioctls.h
 38       socket.ioctl 0x8927, ifreq
 39       ifreq[18, 6] #in scratchpad's raw_ethernet, then says .unpack('H*').first
 40     else 
 41       raise "Unsupported platform #{RUBY_PLATFORM}"
 42     end
 43   end

I changed it so that it prints out the mac: (line numbers don't match)

28       # /usr/include/net/if.h, structure ifreq
 29       ifreq = [eth_device].pack 'a32'
 30       puts "\ndevice name: "
 31       puts ifreq
 32       # 0x8927 is SIOCGIFHWADDR in /usr/include/bits/ioctls.h
 33       socket.ioctl 0x8927, ifreq
 34       puts "mac address output: "
 35       puts ifreq[18, 6].unpack('H*')[0]
 36       puts "-----------"
 37       ifreq[18, 6]

$ sudo rspec raw_socket_spec.rb

device name:
eth0
mac address output:
002170925bc5
-----------
.
device name:
eth0
mac address output:
002170925bc5
-----------
..
device name:
eth0
mac address output:
002170925bc5
-----------
..

Finished in 0.30495 seconds
5 examples, 0 failures

yup, the mac address of my eth0 is indeed that. (matched HWaddr in ifconfig -a of eth0)

the ".unpack('H*')[0]" part changes the mac address into hex, without it, it would look like:
!p�[


hmmm.... so let's see if it gets the right mac address if I change the device to vmnet1
so, raw_socket_spec.rb:

3 describe 'RawSocket' do
  4   let(:eth_device) { 'vmnet1' }

$ sudo rspec raw_socket_spec.rb

device name:
vmnet1
mac address output:
005056c00001
-----------
.
device name:
vmnet1
mac address output:
005056c00001
-----------
..
device name:
vmnet1
mac address output:
005056c00001
-----------

a match! yipee!!! (the rspec hung, but that we don't care)

Also worked for wlan0, even though I'm not plugged in to it

device name:
wlan0
mac address output:
001c26640fac
-----------


============================================

So now, I need just get the list of names and then run .mac() for each of them.
I need to look at the code for systemgetifaddr

man 3 getifaddrs
http://comments.gmane.org/gmane.os.cygwin/123705 ioctl vs getifaddrs

The rspec of https://github.com/bbcoimbra/system-getifaddrs/blob/master/spec/system-ifaddrs_spec.rb compares it to ifconfig's result

added to raw_socket.rb:

2 require 'rubygems' # for listing
  3 require 'system/getifaddrs' # for listing
...
 45   # listing all network devices's MAC addresses
 46   def self.list_all_devices()
 47     case RUBY_PLATFORM
 48     when /linux/ #test to see if works on MacOS as well 
 49       devicehash = System.get_ifaddrs # {:eth0=>{:inet_addr=>"18.248.7.89", :netmask=>"18.    248.7.89"}, :vmnet1=>{:inet_addr=>"172.16.76.1", :netmask=>"172.16.76.1"}}
 50       devicehash.each_key do |key|
 51         puts self.mac(key.to_s).unpack('H*')[0]
 52         puts " - - - - -"
 53       end
 54     end
 55   end

added to raw_socket_spec.rb:

42   describe 'testing listing all network devices' do
 43      it 'should list the devices' do
 44          EtherShell::RawSocket.list_all_devices().should_not be_empty
 45      end
 46   end

good results:

$ sudo rspec  raw_socket_spec.rb 

device name: 
eth0
mac address output: 
002170925bc5
-----------
.
device name: 
eth0
mac address output: 
002170925bc5
-----------
..
device name: 
eth0
mac address output: 
002170925bc5
-----------
..
device name: 
vmnet1
mac address output: 
005056c00001
-----------
005056c00001
 - - - - -

device name: 
vmnet8
mac address output: 
005056c00008
-----------
005056c00008
 - - - - -

device name: 
lo
mac address output: 
000000000000
-----------
000000000000
 - - - - -

device name: 
eth0
mac address output: 
002170925bc5
-----------
002170925bc5
 - - - - -
.

Finished in 0.21653 seconds
6 examples, 0 failures

now I have to make it into 2 functions, one the lists the devices, one that lists their mac addresses (see way top of blog)

Making rspec working on my own computer

Added ethernet.rb  and made rspec work.

To run the rspec, eg:

sudo rspec socket_wrapper_spec.rb

...............

Finished in 7.61 seconds
15 examples, 0 failures

CANNOT be using Wireless connection for the rspec to work. MUST be using Ethernet.
commit c21eb650831d6921e33fa393e5f7d3cd3fdd2d1e

Boring details of how I got here

RawSocket mac should have 6 bytes
Failure/Error: let(:mac) { EtherShell::RawSocket.mac eth_device }
Errno::EPERM:
Operation not permitted - socket(2)

man 2 socket

SOCKET(2)                        Linux Programmer's Manual                       SOCKET(2)

NAME
       socket - create an endpoint for communication

SYNOPSIS
       #include           /* See NOTES */
       #include 

       int socket(int domain, int type, int protocol);

DESCRIPTION
       socket() creates an endpoint for communication and returns a descriptor.

       The  domain  argument  specifies  a communication domain; this selects the protocol
       family which will be  used  for  communication.   These  families  are  defined  in
       <sys/socket.h>.  The currently understood formats include:

       Name                Purpose                          Man page
       AF_UNIX, AF_LOCAL   Local communication              unix(7)
       AF_INET             IPv4 Internet protocols          ip(7)
       AF_INET6            IPv6 Internet protocols          ipv6(7)
       AF_IPX              IPX - Novell protocols
       AF_NETLINK          Kernel user interface device     netlink(7)
       AF_X25              ITU-T X.25 / ISO-8208 protocol   x25(7)
       AF_AX25             Amateur radio AX.25 protocol
       AF_ATMPVC           Access to raw ATM PVCs
       AF_APPLETALK        Appletalk                        ddp(7)
       AF_PACKET           Low level packet interface       packet(7)

..../ether_shell/spec/ether_shell$sudo rspec raw_socket_spec.rb
[sudo] password for haoqili:
.....

Finished in 0.06603 seconds
5 examples, 0 failures

/ethernet/spec$ sudo rspec raw_socket_spec.rb 
[sudo] password for haoqili: 
/usr/lib/ruby/gems/1.8/gems/rspec-core-2.5.1/lib/rspec/core/backward_compatibility.rb:20:in `const_missing': uninitialized constant ShellStub::EtherShell (NameError)
 from /usr/lib/ruby/gems/1.8/gems/rspec-expectations-2.5.0/lib/rspec/expectations/backward_compatibility.rb:6:in `const_missing'
 from /home/haoqili/Desktop/rails/ethernet/spec/support/shell_stub.rb:2
 from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
 from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
 from /home/haoqili/Desktop/rails/ethernet/spec/spec_helper.rb:8
 from /home/haoqili/Desktop/rails/ethernet/spec/spec_helper.rb:8:in `each'
 from /home/haoqili/Desktop/rails/ethernet/spec/spec_helper.rb:8
 from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
 from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
 from /home/haoqili/Desktop/rails/ethernet/spec/raw_socket_spec.rb:3
 from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `load'
 from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `load_spec_files'
 from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `map'
 from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `load_spec_files'
 from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.5.1/lib/rspec/core/command_line.rb:18:in `run'
 from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.5.1/lib/rspec/core/runner.rb:55:in `run_in_process'
 from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.5.1/lib/rspec/core/runner.rb:46:in `run'
 from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.5.1/lib/rspec/core/runner.rb:10:in `autorun'
 from /usr/bin/rspec:19

In ./spec/raw_socket_spec.rb, ./spec/socket_wrapper_spec.rb
:%s/EtherShell/Ethernet/g
Didn't fix


After talking to Victor, he told me that I don't have ethernet.rb, the equivalent of ether_shell.rb. So modified ether_shell.rb into ethernet.rb.

Fixed.

system-getifaddrs

The creator of system-getifaddrs, Bruno Coimbra, is so awesome! He helped me fix the problems I had here.

Setting up system-getifaddrs:

Inside the system-getifaddrs directory, I did:

$ sudo gem install system-getifaddrs

[sudo] password for haoqili:
Fetching: system-getifaddrs-0.1.1.gem (100%)
Building native extensions. This could take a while...
Successfully installed system-getifaddrs-0.1.1
1 gem installed
Installing ri documentation for system-getifaddrs-0.1.1...
Installing RDoc documentation for system-getifaddrs-0.1.1...

Then I did:
$ irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'system/getifaddrs'
=> true
irb(main):003:0> p System.get_ifaddrs

{:wlan0=>{:inet_addr=>"18.111.61.140", :netmask=>"18.111.61.140"}, :vmnet1=>{:inet_addr=>"172.16.76.1", :netmask=>"172.16.76.1"}, :vmnet8=>{:inet_addr=>"172.16.197.1", :netmask=>"172.16.197.1"}, :lo=>{:inet_addr=>"127.0.0.1", :netmask=>"127.0.0.1"}}
=> nil

me: how come eth0 is not in the list?
Bruno: are you connected on wireless?
me: I'm on wireless
Bruno: you are using
           eth0 should be down


Making test.rb work. Add the first line, so it's like:
1 require "rubygems"
2 require "pp"
3 require "system/getifaddrs"
4 pp System.get_ifaddrs

Where pp is for "pretty print".
$ ruby test.rb

{:wlan0=>{:inet_addr=>"18.111.61.140", :netmask=>"18.111.61.140"},
:vmnet1=>{:inet_addr=>"172.16.76.1", :netmask=>"172.16.76.1"},
:vmnet8=>{:inet_addr=>"172.16.197.1", :netmask=>"172.16.197.1"},
:lo=>{:inet_addr=>"127.0.0.1", :netmask=>"127.0.0.1"}}

Without the first line, the result would be as below. I need the "rubygems" because I am running system-wide ruby instead of rvm, changed to it here. (which ruby, gives "/usr/bin/ruby")

test.rb:2:in `require': no such file to load -- system/getifaddrs (LoadError)
from test.rb:2

Where is the *.gem file?
me: is system-getifaddrs a ruby gem?
Bruno: yep
me: how come it doesn't have a *.gem file anywhere?
Bruno: ah, on github have only source files
me: so the *.gem file is not necessary for it to run?
Bruno: *.gem file is the package file like deb packages for debian
             after installing, files are unpacked under ruby lib dir and gem file is discarted
me: so, on github you put the state after unpackaging, that's why .gem doesn't need to be there
Bruno: nope, on github is development sources. with that sources you can packaging its into a gem file

Getting ethernet_ping to work

Goal: recreate scratchpad's ether_ping in my ethernet gem.

AF / PF POSIX Socket API

Ruby learning
ruby array pack() array-->binary string
string unpack string --> array
array first()
attr_reader :to_return is an alias for returning that instance's to_return.

ether_ping  eth0                88B7                       001122334455        aa00112233
ether_ping  net_interface   ether_type 2bytes   dest_mac 6byte        data 0pad to 64 bytes

scratchpad's ether_ping, relies on
- scratchpad/bin/
        - ether_ping is the client
                  client = Scratchpad::Ethernet::PingClient.new interface, ether_type, dest_mac
                  puts client.ping(data)
        - ether_ping_server
                  it seems that calling ether_ping doesn't need ether_ping_server's stuff
- scratchpad/lib/scratchpad/ethernet
        - ping.rb has classes: PingServer and PingClient
                 has require 'eventmachine' for communication
        - raw_ethernet.rb connects to the raw_ethernet stuff

What I did:
1. keep looking at that. Especially socket http://www.ruby-doc.org/stdlib/libdoc/socket/rdoc/classes/Socket.html
2. ethernet is basically ether_shell renamed, with scratchpad's pinging
3. Put in system-getifaddr in too

trying

ethernet$ rake
(in /home/haoqili/Desktop/rails/ethernet)
/usr/bin/ruby1.8 -S bundle exec rspec "spec/raw_socket_spec.rb" "spec/ethernet_spec.rb" "spec/socket_wrapper_spec.rb"
/usr/lib/ruby/gems/1.8/gems/rspec-core-2.3.1/lib/rspec/core/backward_compatibility.rb:20:in `const_missing': uninitialized constant ShellStub::Ethernet (NameError)
from /usr/lib/ruby/gems/1.8/gems/rspec-expectations-2.3.0/lib/rspec/expectations/backward_compatibility.rb:6:in `const_missing'
from /home/haoqili/Desktop/rails/ethernet/spec/support/shell_stub.rb:2
from /home/haoqili/Desktop/rails/ethernet/spec/spec_helper.rb:8:in `require'
from /home/haoqili/Desktop/rails/ethernet/spec/spec_helper.rb:8
from /home/haoqili/Desktop/rails/ethernet/spec/spec_helper.rb:8:in `each'
from /home/haoqili/Desktop/rails/ethernet/spec/spec_helper.rb:8
from /home/haoqili/Desktop/rails/ethernet/spec/raw_socket_spec.rb:4:in `require'
from /home/haoqili/Desktop/rails/ethernet/spec/raw_socket_spec.rb:4
from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.3.1/lib/rspec/core/configuration.rb:388:in `load'
from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.3.1/lib/rspec/core/configuration.rb:388:in `load_spec_files'
from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.3.1/lib/rspec/core/configuration.rb:388:in `map'
from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.3.1/lib/rspec/core/configuration.rb:388:in `load_spec_files'
from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.3.1/lib/rspec/core/command_line.rb:18:in `run'
from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.3.1/lib/rspec/core/runner.rb:55:in `run_in_process'
from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.3.1/lib/rspec/core/runner.rb:46:in `run'
from /usr/lib/ruby/gems/1.8/gems/rspec-core-2.3.1/lib/rspec/core/runner.rb:10:in `autorun'
from /usr/bin/rspec:19
rake aborted!
ruby -S bundle exec rspec "spec/raw_socket_spec.rb" "spec/ethernet_spec.rb" "spec/socket_wrapper_spec.rb" failed

(See full trace by running task with --trace)

ethernet$ sudo rake install
(in /home/haoqili/Desktop/rails/ethernet)
Successfully built RubyGem
Name: ethernet
Version: 0.0.0
File: ethernet-0.0.0.gem
Executing "ruby1.8 -S gem install ./pkg/ethernet-0.0.0.gem":
ruby1.8 -S gem install ./pkg/ethernet-0.0.0.gem
Successfully installed ethernet-0.0.0
1 gem installed
Installing ri documentation for ethernet-0.0.0...
Installing RDoc documentation for ethernet-0.0.0...


========
added "s.executables =["ethernet_ping"]" in ethernet.gemspec

/ethernet$ sudo rake install
[sudo] password for haoqili:
(in /home/haoqili/Desktop/rails/ethernet)
Successfully built RubyGem
Name: ethernet
Version: 0.0.0
File: ethernet-0.0.0.gem
Executing "ruby1.8 -S gem install ./pkg/ethernet-0.0.0.gem":
ruby1.8 -S gem install ./pkg/ethernet-0.0.0.gem
Building native extensions. This could take a while...
Successfully installed eventmachine-0.12.10
Successfully installed ethernet-0.0.0
2 gems installed
Installing ri documentation for eventmachine-0.12.10...
Installing ri documentation for ethernet-0.0.0...
Installing RDoc documentation for eventmachine-0.12.10...
Could not find main page README
Could not find main page README
Could not find main page README
Could not find main page README
Installing RDoc documentation for ethernet-0.0.0...


still not found
ethernet$ ethernet_ping
ethernet_ping: command not found

deleted s.executables = ["ethernet_ping"] from ethernet.gemspec
added gem.executables = ["ethernet_ping"] to Rakefile
$ sudo rake install
[sudo] password for haoqili:
(in /home/haoqili/Desktop/rails/ethernet)
Successfully built RubyGem
Name: ethernet
Version: 0.0.0
File: ethernet-0.0.0.gem
Executing "ruby1.8 -S gem install ./pkg/ethernet-0.0.0.gem":
ruby1.8 -S gem install ./pkg/ethernet-0.0.0.gem
Building native extensions. This could take a while...
Successfully installed eventmachine-0.12.10
Successfully installed ethernet-0.0.0
2 gems installed
Installing ri documentation for eventmachine-0.12.10...
Installing ri documentation for ethernet-0.0.0...
Installing RDoc documentation for eventmachine-0.12.10...
Could not find main page README
Could not find main page README
Could not find main page README
Could not find main page README
Installing RDoc documentation for ethernet-0.0.0...

haoqili@stay-focused-hq:18:32:11:~/Desktop/rails/ethernet$ ethernet_ping
Usage: /usr/bin/ethernet_ping net_interface ether_type dest_mac data
net_interface: name of the Ethernet interface, e.g. eth0
ether_type: packet type for the Ethernet II frame, in hex (2 bytes)
dest_mac: destination MAC for the ping packets, in hex (6 bytes)
data: ping packet data, in hex (0-padded to 64 bytes)

haoqili@stay-focused-hq:18:35:52:~/Desktop/rails/ethernet$ sudo ethernet_ping eth0 88B7 001122334455 aa00112233
/usr/lib/ruby/gems/1.8/gems/ethernet-0.0.0/bin/ethernet_ping:22: uninitialized constant Ethernet (NameError)
from /usr/bin/ethernet_ping:19:in `load'
from /usr/bin/ethernet_ping:19

19 load Gem.bin_path('ethernet', 'ethernet_ping', version)

That doesn't look right

* After Victor told me to add ethernet.rb (and then I did sudo rake install), the Ethernet NameError is gone, but now I have

$ sudo ethernet_ping eth0 88B7 001122334455 aa00112233
[sudo] password for haoqili:
/usr/lib/ruby/gems/1.8/gems/ethernet-0.0.0/bin/ethernet_ping:22: uninitialized constant Ethernet::PingClient (NameError)
from /usr/bin/ethernet_ping:19:in `load'
from /usr/bin/ethernet_ping:19

* Ah, because I never put in the file about ethernet pinging in "ethernet.rb", so I added this line:
require 'ping.rb' # This is to get ethernet_ping working :P

sudo rake install again
$ sudo ethernet_ping eth0 88B7 001122334455 aa00112233
/usr/lib/ruby/gems/1.8/gems/ethernet-0.0.0/lib/ping.rb:63:in `initialize': undefined method `socket' for Ethernet:Module (NoMethodError)
from /usr/lib/ruby/gems/1.8/gems/ethernet-0.0.0/bin/ethernet_ping:22:in `new'
from /usr/lib/ruby/gems/1.8/gems/ethernet-0.0.0/bin/ethernet_ping:22
from /usr/bin/ethernet_ping:19:in `load'
from /usr/bin/ethernet_ping:19

This is because I put my "socket" method inside the RawSocket module as well, so for methods ".socket" and ".mac", I have to change their prefix from "Ethernet::" to "Ethernet::RawSocket".


I also reordered where the ruby files are stored. Victor told me that the way he set ether_shell up is how it's done by convention. "IDEs have shortcuts that let you navigate between code and spec, as long as things stay where they're supposed to be."

so after "sudo rake install", I tried the above ethernet_ping, but I got another error:

$ sudo ethernet_ping eth0 88B7 001122334455 aa00112233
/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require': no such file to load -- ethernet/raw_socket.rb (LoadError)
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
from /usr/lib/ruby/gems/1.8/gems/ethernet-0.0.0/lib/ethernet.rb:6
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
from /usr/lib/ruby/gems/1.8/gems/ethernet-0.0.0/bin/ethernet_ping:3
from /usr/bin/ethernet_ping:19:in `load'
from /usr/bin/ethernet_ping:19

This is at commit 1b6daf320e82219a15d52d734dcbb10aff1b5fb6.

The next day, I went to see Victor. He told me to check:

$ ls /usr/lib/ruby/gems/1.8/gems/ethernet-0.0.0/lib
ethernet.rb

Indeed, the ethernet dir was not there.

I did again:

ethernet$ sudo rake install
(in /home/haoqili/Desktop/rails/ethernet)
  Successfully built RubyGem
  Name: ethernet
  Version: 0.0.0
  File: ethernet-0.0.0.gem
Executing "ruby1.8 -S gem install ./pkg/ethernet-0.0.0.gem":
ruby1.8 -S gem install ./pkg/ethernet-0.0.0.gem
Building native extensions.  This could take a while...
Successfully installed eventmachine-0.12.10
Successfully installed ethernet-0.0.0
2 gems installed
Installing ri documentation for eventmachine-0.12.10...
Installing ri documentation for ethernet-0.0.0...
Installing RDoc documentation for eventmachine-0.12.10...
Could not find main page README
Could not find main page README
Could not find main page README
Could not find main page README

$ ls /usr/lib/ruby/gems/1.8/gems/ethernet-0.0.0/lib
ethernet  ethernet.rb
haoqili@stay-focused-hq:15:16:56:~/Desktop/rails/ethernet$ sudo ethernet_ping eth0 88B7 001122334455 aa00112233
Pinging 001122334455... 

Fixed!

Shoot. It didn't show up on Wireshark :/

I printed out the source mac, sure it doesn't match eth0's mac address:
eth0 Link encap:Ethernet HWaddr 00:21:70:92:5b:c5

57 class PingClient
 58   def initialize(if_name, ether_type, destination_mac)
 59     @socket = Ethernet.socket if_name, ether_type
 60 
 61     @source_mac = [Ethernet.get_interface_mac(if_name)].pack('H*')[0, 6]
 62     puts "\n the source_mac, is"
 63     puts @source_mac.unpack('H*')[0]

ethernet's result:

$ sudo ethernet_ping eth0 88B7 001122334455 aa00112233

 the source mac is:
0192b5
Pinging 001122334455... 

scratchpad's result: (scratchpad has the working ether_ping)

$ sudo ether_ping eth0 88B7 001122334455 aa00112233

 the source_mac, is
002170925bc5
Pinging 001122334455... 

This is because ether_shell's raw_socket's .mac() doesn't have "unpack('H*').first", so if I put this back in, pinging would work in ethernet

raw_socket.rb
 32   def self.mac(eth_device)
 33     case RUBY_PLATFORM
 34     when /linux/
 35       # /usr/include/net/if.h, structure ifreq
 36       ifreq = [eth_device].pack 'a32'
 37       # 0x8927 is SIOCGIFHWADDR in /usr/include/bits/ioctls.h
 38       socket.ioctl 0x8927, ifreq
 40       #ifreq[18, 6] #in scratchpad's raw_ethernet, then says .unpack('H*').first
 41                    # because raw socket should return raw data
 42                    # leaving higher layers for presentation issues
 43     else

add it in here instead

ping.rb
66     @source_mac = [Ethernet::RawSocket.mac(eth_device).unpack('H*').first].pack('H*')[0, 6]

Yay worked!

commit: bff1c0ca79ed679aae77e5e6199c2aaf3e75fdc9

:) ethernet_ping works! Try with 'sudo ethernet_ping eth0 88B7 001122334455 aa00112233' and see it in Wireshark's eth0

How Wireshark lists devices

Goal: find how Wireshark gets the list of MAC addresses at the beginning (Windows, MacOS)
Edit: Later found out that neither Wireshark nor libpcap gives a list of MAC addresses.
Answer: To get a list of internet device names, Wireless uses dumpcap's pcap_findalldevs().

I. I looked for MAC address
II. I looked for just the simple list of devices Wireshark shows

I. I grep-ed for "MAC address" and found over 500 entries, under these big categories：
- epan/dissectors (huge)
- plugins
- airpcap

Interesting things found:

• Ulf Lamping, ulf.lamping[at]web.de, put the source and destination MAC addresses into the top-level item for Ethernet.
• "manuf" contains Ethernet vendor codes (i.e. the first half of MAC Addresses), and well-known MAC addresses. So I checked the MAC Addresses from my ifconfig -a. eth0's first half is 00:21:70, which manuf says is Dell (yes I have a Dell computer). vmnet*'s first half is 00:50:56, which manuf says is VMware. wlan0's is 00:1c:26, which manuf says is Hon Hai Precision Ind. Co. Cool.
  
• Looking at epan/dissectors doesn't seem right. Hmm, 
• Talked to Victor about it, he told me to look at the RubyGem (see other post) first

II. Give up. So what if I just track it down? The list at the beginning of Wireshark appears below the words "Start capture on interface:". So I grep-ed for those words and found that it appears in ./gtk/main_welcome.c.

There are 2 ways to go in main_welcome.c
1. WIRESHARK_STOCK_CAPTURE_INTERFACES from clicking on "Interface List" button on the main page. This is from

812 welcome_button(WIRESHARK_STOCK_CAPTURE_INTERFACES,
813         "Interface List",
814         "Live list of the capture interfaces\n(counts incoming packets)", ...

But searching down this path wasn't very fruitful

2a. By looking at how the list of devices is generated directly with the welcome_if_panel_load() function.

653 /* load the list of interfaces */
654 static void
655 welcome_if_panel_load(void)

661     GList         *if_list;
662     int err;
663     gchar         *err_str = NULL;


669     /* LOAD THE INTERFACES */

670     if_list = capture_interface_list(&err, &err_str);

btw, the g_blah() functions come from GLib

2b. So I now grep-ed for capture_interface_list(), which attempts to open all adapters it finds in order to check whether they can be captured on.

capture_interface_list() in ./capture_ifinfo.c

 1 /* capture_ifinfo.c
  2  * Routines for getting interface information from dumpcap
 23  */
 24
 25 #ifdef HAVE_CONFIG_H
 26 # include "config.h"
 27 #endif
 28
 29 #ifdef HAVE_LIBPCAP
 30
 31 #include <stdlib.h>
 32 #include <string.h>
 33
 34 #ifdef HAVE_ARPA_INET_H
 35 #include <arpa/inet.h>
 36 #endif
 37
 38 #ifdef HAVE_SYS_SOCKET_H
 39 #include <sys/socket.h>         /* needed to define AF_ values on UNIX */
 40 #endif
 41
 42 #ifdef HAVE_WINSOCK2_H
 43 #include <winsock2.h>           /* needed to define AF_ values on Windows */

 44 #endif

 45 

 46 #ifdef NEED_INET_V6DEFS_H

 47 # include "wsutil/inet_v6defs.h"

 48 #endif

 49 

 50 #include <glib.h>

 51 

 52 #include "capture_opts.h"

 53 #include "capture_sync.h"

 54 #include "log.h"

 55 

 56 #include "capture_ifinfo.h"

 57 

 58 /**

 59  * Fetch the interface list from a child process (dumpcap).

 60  *

 61  * @return A GList containing if_info_t structs if successful, NULL (with err and possibly err_str set) otherwise.

 62  *

 63  */

 64 

 65 /* XXX - We parse simple text output to get our interface list.  Should

 66  * we use "real" data serialization instead, e.g. via XML? */

 67 GList *

 68 capture_interface_list(int *err, char **err_str)

 69 {

 70     int        ret;

 71     GList     *if_list = NULL;

 72     int        i, j;

 73     gchar     *data, *primary_msg, *secondary_msg;

 74     gchar    **raw_list, **if_parts, **addr_parts;

 75     gchar     *name;

 76     if_info_t *if_info;

 77     if_addr_t *if_addr;

 78 

 79     g_log(LOG_DOMAIN_CAPTURE, G_LOG_LEVEL_MESSAGE, "Capture Interface List ...");

 80 

 81     /* Try to get our interface list */

 82     ret = sync_interface_list_open(&data, &primary_msg, &secondary_msg);

 83     if (ret != 0) {

 84         g_log(LOG_DOMAIN_CAPTURE, G_LOG_LEVEL_MESSAGE, "Capture Interface List failed!");

 85         if (err_str) {

 86             *err_str = primary_msg;

 87         } else {

 88             g_free(primary_msg);

 89         }

 90         g_free(secondary_msg);

 91         *err = CANT_GET_INTERFACE_LIST;

 92         return NULL;

 93     }

 94 

 95     /* Split our lines */

 96 #ifdef _WIN32

 97     raw_list = g_strsplit(data, "\r\n", 0);

 98 #else

 99     raw_list = g_strsplit(data, "\n", 0);

100 #endif

101     g_free(data);

102 

103     for (i = 0; raw_list[i] != NULL; i++) {

104         if_parts = g_strsplit(raw_list[i], "\t", 4);

105         if (if_parts[0] == NULL || if_parts[1] == NULL || if_parts[2] == NULL ||

106                 if_parts[3] == NULL) {

107             g_strfreev(if_parts);

108             continue;

109         }

110 

111         /* Number followed by the name, e.g "1. eth0" */

112         name = strchr(if_parts[0], ' '); /*strchr searches to see if there is a single
                                                                space chr in if_parts[0] */

113         if (name) {

114             name++;

115         } else {

116             g_strfreev(if_parts);

117             continue;

118         }

119 

120         if_info = g_malloc0(sizeof(if_info_t));

121         if_info->name = g_strdup(name);

122         if (strlen(if_parts[1]) > 0)

123             if_info->description = g_strdup(if_parts[1]);

124         addr_parts = g_strsplit(if_parts[2], ",", 0);

125         for (j = 0; addr_parts[j] != NULL; j++) {

126             if_addr = g_malloc0(sizeof(if_addr_t));

127             if (inet_pton(AF_INET, addr_parts[j], &if_addr->addr.ip4_addr)) {

128                 if_addr->ifat_type = IF_AT_IPv4;

129             } else if (inet_pton(AF_INET6, addr_parts[j],

130                     &if_addr->addr.ip6_addr)) {

131                 if_addr->ifat_type = IF_AT_IPv6;

132             } else {

133                 g_free(if_addr);

134                 if_addr = NULL;

135             }

136             if (if_addr) {

137                 if_info->addrs = g_slist_append(if_info->addrs, if_addr);

138             }

139         }

140         if (strcmp(if_parts[3], "loopback") == 0)

141             if_info->loopback = TRUE;

142         g_strfreev(if_parts);

143         g_strfreev(addr_parts);

144         if_list = g_list_append(if_list, if_info);

145     }

146     g_strfreev(raw_list);

147 

148     /* Check to see if we built a list */

149     if (if_list == NULL) {

150         *err = NO_INTERFACES_FOUND;

151         if (err_str)

152             *err_str = g_strdup("No interfaces found");

153     }

154     return if_list;

155 }

2c. Then look at: sync_interface_list_open(), which is found in ./capture_sync.c:

1073 /*
1074  * Get the list of interfaces using dumpcap.
1075  *
1076  * On success, *data points to a buffer containing the dumpcap output,
1077  * *primary_msg and *secondary_msg are NULL, and 0 is returned.  *data
1078  * must be freed with g_free().
1079  *
1080  * On failure, *data is NULL, *primary_msg points to an error message,
1081  * *secondary_msg either points to an additional error message or is
1082  * NULL, and -1 is returned; *primary_msg, and *secondary_msg if not NULL,
1083  * must be freed with g_free().
1084  */
1085 int
1086 sync_interface_list_open(gchar **data, gchar **primary_msg,
1087                          gchar **secondary_msg)
1088 {
1089     int argc;
1090     const char **argv;
1091
1092     g_log(LOG_DOMAIN_CAPTURE, G_LOG_LEVEL_DEBUG, "sync_interface_list_open");
1093
1094     argv = init_pipe_args(&argc);
1095
1096     if (!argv) {
1097         *primary_msg = g_strdup("We don't know where to find dumpcap.");
1098         *secondary_msg = NULL;
1099         *data = NULL;
1100         return -1;
1101     }
1102
1103     /* Ask for the interface list */
1104     argv = sync_pipe_add_arg(argv, &argc, "-D");
1105
1106 #ifndef DEBUG_CHILD
1107     /* Run dumpcap in capture child mode */
1108     argv = sync_pipe_add_arg(argv, &argc, "-Z");
1109     argv = sync_pipe_add_arg(argv, &argc, SIGNAL_PIPE_CTRL_ID_NONE);
1110 #endif
1111     return sync_pipe_run_command(argv, data, primary_msg, secondary_msg);
1112 }

It's passing in an argument to get the devices, and the argument is "dumpcap"

2d. dumpcap and dumpcap.c
dumpcap is cross platform. :)
man dumpcap told me that the option "-D" prints a list of the interfaces on which Dumpcap can capture.

I found dumpcap's code in "dumpcap.c":

3792      * "-D" requires no interface to be selected; it's supposed to list
3793      * all interfaces.
3794      */
3795     if (list_interfaces) {
3796         /* Get the list of interfaces */
3797         GList       *if_list;
3798         int         err;
3799         gchar       *err_str;
3800 
3801         if_list = capture_interface_list(&err, &err_str);

So then I track down this function

796 GList *
 797 capture_interface_list(int *err, char **err_str)
 798 {
 799     return get_interface_list(err, err_str);
 800 }

after "grep -ir get_interface_list ." I found it in:

2e. capture-wpcap.c

676 /*
677  * This will use "pcap_findalldevs()" if we have it, otherwise it'll
678  * fall back on "pcap_lookupdev()".
679  */
680 GList *
681 get_interface_list(int *err, char **err_str)
682 {
683     GList  *il = NULL;
684     wchar_t *names;
685     char *win95names;
686     char ascii_name[MAX_WIN_IF_NAME_LEN + 1];
687     char ascii_desc[MAX_WIN_IF_NAME_LEN + 1];
688     int i, j;
689     char errbuf[PCAP_ERRBUF_SIZE];
690 
691 #ifdef HAVE_PCAP_FINDALLDEVS
692     if (p_pcap_findalldevs != NULL)
693         return get_interface_list_findalldevs(err, err_str);
694 #endif
...
738 
739     names = (wchar_t *)pcap_lookupdev(errbuf);

There are 2 methods, get_interface_list_findalldevs() or pcap_lookupdev()

Both methods will require code from libpcap, which I found on this site.

Working out the 2 different ways

2d.1. First, let's try with get_interface_list_findalldevs()
Inside capture-pcap-util.c

107 #ifdef HAVE_PCAP_FINDALLDEVS
...
166 GList *
167 get_interface_list_findalldevs(int *err, char **err_str)
168 {
169     GList  *il = NULL;
170     pcap_if_t *alldevs, *dev;
171     if_info_t *if_info;
172     char errbuf[PCAP_ERRBUF_SIZE];
173 
174     if (pcap_findalldevs(&alldevs, errbuf) == -1) {
175         *err = CANT_GET_INTERFACE_LIST;
176         if (err_str != NULL)
177             *err_str = cant_get_if_list_error_message(errbuf);
178         return NULL;
179     }
180 
181     if (alldevs == NULL) {
182         /*
183          * No interfaces found.
184          */
185         *err = NO_INTERFACES_FOUND;
186         if (err_str != NULL)
187             *err_str = NULL;
188         return NULL;
189     }
190 
191     for (dev = alldevs; dev != NULL; dev = dev->next) {
192         if_info = if_info_new(dev->name, dev->description);
193         il = g_list_append(il, if_info);
194         if_info_ip(if_info, dev);
195     }
196     pcap_freealldevs(alldevs);
197 
198     return il;
199 }
200 #endif /* HAVE_PCAP_FINDALLDEVS */

2d.2 Now, let's try with the other method pcap_lookupdev()

233 char*
234 pcap_lookupdev (char *a)
235 {
236     if (!has_wpcap) {
237         return NULL;
238     }
239     return p_pcap_lookupdev(a);
240 }

which is a pointer

56 static char*   (*p_pcap_lookupdev) (char *);

2d.2b. Now we have to look at libpcap's code for pcap_lookupdev, which will also lead to

pcap_findalldevs()

Found it in inet.c

635 #if !defined(WIN32) && !defined(MSDOS)
636 
637 /*
638  * Return the name of a network interface attached to the system, or NULL
639  * if none can be found.  The interface must be configured up; the
640  * lowest unit number is preferred; loopback is ignored.
641  */
642 char *
643 pcap_lookupdev(errbuf)
644     register char *errbuf;
645 {
...
654     if (pcap_findalldevs(&alldevs, errbuf) == -1)

777 #elif defined(WIN32)
778 
779 /*
780  * Return the name of a network interface attached to the system, or NULL
781  * if none can be found.  The interface must be configured up; the
782  * lowest unit number is preferred; loopback is ignored.
783  */
784 char *
785 pcap_lookupdev(errbuf)
786     register char *errbuf;
787 {
788     DWORD dwVersion;
789     DWORD dwWindowsMajorVersion;
790     dwVersion = GetVersion();   /* get the OS version */
791     dwWindowsMajorVersion = (DWORD)(LOBYTE(LOWORD(dwVersion)));
792 
793     if (dwVersion >= 0x80000000 && dwWindowsMajorVersion >= 4) {
794         /*
795          * Windows 95, 98, ME.
796          */
797         ULONG NameLength = 8192;
798         static char AdaptersName[8192];
799 
800         if (PacketGetAdapterNames(AdaptersName,&NameLength) )
801             return (AdaptersName);
802         else
803             return NULL;
804     } else {
805         /*
806          * Windows NT (NT 4.0, W2K, WXP). Convert the names to UNICODE for backward compatibiliy

3. Now we need to look inside libpcap-1.1.1 for pcap_findalldevs()

Method description from pcap-sita.html:
SMP The Supervisory Management Processor where Wireshark (or equivalent) runs in conjuction with a libpcap front-end.
IOP I/O Processors where the monitored ports exist in conjunction with a custom device driver/libpcap back-end.

pcap_findalldevs constructs a list of network devices that can be opened with pcap_open_live().

SMP It obtains a list of IOPs currently available (via /etc/hosts).
SMP -> IOP The SMP will sequentially open a connection to each IOP on its 'sniffer' port to ensure the IOP is available. It sends a null terminated empty interface ID followed by the query request command.
IOP -> SMP The IOP returns an error response and its list of devices.
SMP -> IOP The SMP closes the TCP connection with each IOP.
SMP The SMP adds the received information to its internal structure.

=========================
SMP/IOP Inter-Process Communication Protocol

Communications between an SMP and an IOP consists of a TCP session
between an ephemeral port on the SMP and the well known port of 49152
(which is the first available port in the 'dynamic and/or private port'
range) on an IOP.

Following a TCP open operation the IOP receives a null terminated
'interface ID' string to determine the type of operation that follows:

Every command received by an IOP implies a 'stop trace/stop forwarding' operation must
occur before executing the received command.

A session is closed when the SMP closes the TCP session with the IOP.
Obviously monitoring and forwarding is also stopped at that time.

Note: All multi-octet entities are sent in network neutral order.

SMP -> IOP	Open socket (to each IOP), and sends: Name/ Purpose Size (in bytes) Description Interface ID 1 A NULL to indicate an an empty 'interface ID'.
IOP -> SMP	Send its (possibly empty) NULL terminated error response string.
SMP -> IOP	Sends the 'interface query request': Name/ Purpose Size (in bytes) Description Interface ID 1 A 'Q' (indicating 'interface query request').
IOP -> SMP	The IOP returns a list of sequences of information as defined by the return parameter of this function call (as shown in the following table). Elements are specified by providing an unsigned byte preceeding the actual data that contains length information. Notes: Name/ Purpose Size (in bytes) Description length 1 The number of octets in the name field that follows. Name 1-255 The name of the interface. The format of the name is an alphabetic string (indicating the type of interface) followed by an optional numeric string (indicating the interface's sequence number). Sequence numbers (if needed) will begin at zero and progress monotonically upwards. (i.e. 'eth0', 'lo', 'wan0', etc.) For an IOP, the alphabetic string will be one of: 'eth', 'wan', and 'lo' for Ethernet, WAN ports and the IP loopback device respectively. An IOP currently supports: 'eth0', 'eth1', 'lo', 'wan0' ... 'wan7'. Note: IOPs and ACNs will not currently support the concept of 'any' interface. length 1 The number of octets in the interface description field that follows. Interface Description 0-255 A description of the interface or it may be an empty string. (i.e. 'ALC') Interface Type 4 The type of interface as defined in the description for pcap_datalink() (in network neutral order). Loopback Flag 1 1 = if the interface is a loopback interface, zero = otherwise. count 1 # of address entries that follow. Each entry is a series of bytes in network neutral order. See the parameter definition above for more details. Repeated 'count' number of times. length 1 The number of octets in the address field that follows. Address 1-255 The address of this interface (in network neutral order). length 1 The number of octets in the netmask field that follows. Network Mask 0-255 The network mask used on this interface (if applicable) (in network neutral order). length 1 The number of octets in the broadcast address field that follows. Broadcast Address 0-255 The broadcast address of this interface (if applicable) (in network neutral order). length 1 The number of octets in the destination address field that follows. Destination Address 0-255 The destination address of this interface (if applicable) (in network neutral order).
SMP -> IOP	Close the socket.
IOP -> SMP	Close the socket.

=========================


I have "pcap_findalldevs" source from the following files in libpcap-1.1.1:
they all take the same arguments
fad-win32.c - for Windows OS

216  * Win32 implementation, based on WinPcap
217  */
218 int
219 pcap_findalldevs(pcap_if_t **alldevsp, char *errbuf)

pcap-dos.c - for DOS (predecessor of Windows)
fad-glif.c - for Solaris
fad-gifc.c - This is the implementation used on platforms that have SIOCGIFCONF but don't have any other mechanism for getting a list of interfaces.
fad-getad.c - This is the implementation used on platforms that have "getifaddrs()"

(btw MacOS uses free bsd.)

Already have a gem to get Linux list MAC Address working
MacOS can be on hold
Focus on getting Windows to work! look at fad-win32.c in more detail

MacOS MAC Address

I emailed Victor with this link that gets MAC addresses on MacOS. His reply:


That is one way of going about it, but it's probably very painful. You'd have to make libffi bindings for all those OSX-specific calls.

Google took me here: http://forum.soft32.com/mac/sample-code-MAC-IP-address-Mac-OS-ftopict45416.html

Then I saw the comment at the bottom, and it makes some good points. I googled for a ruby gem that wraps getifaddres, and found this. https://github.com/bbcoimbra/system-getifaddrs

See if you can use it to get the network card information. If it works, you'll probably have to figure out how to build Windows-specific gems, because I'm almost sure this won't install on Windows.

===

So I git cloned "system-getifaddrs".

The problems are fixed here.

Problem 1: I can't the test.rb from its README to work

I found a extconf.rb in system-getifaddrs/ext/rb_getifaddrs, and did:

$ ruby extconf.rb
checking for arpa/inet.h... yes
checking for sys/socket.h... yes
checking for sys/types.h... yes
checking for netdb.h... yes
checking for ifaddrs.h... yes
checking for stdio.h... yes
checking for stdlib.h... yes
checking for unistd.h... yes
creating Makefile

I don't think that's what I'm supposed to do
Problem 2: How do I up "system-getifaddrs" to use it?

Problem 3: If system-getifaddrs is a ruby gem, where is the ".gem" file?

Reading Wireshark

Goal: find how Wireshark gets the MAC address on Windows and MacOS

To reach the goal:
1. finish reading Wireshark docs, see notes below
2. read victor email, see this post
3. read grep of "mac address" in doc. It was useless
4. read grep of "mac address" from wireshark root
5. email wireshark dev
6. How to get Linux MAC address? look in ether_shell

At first, I grep-ed for "MAC address" in the entire Wireshark directory with grep -ir "mac address" . > hq_macaddsearch.txt. It took way to long to complete so I stopped it. Then I looked that it already had over 150,000 lines where the bottom is filled with "hq_macaddsearch.txt" ... oh I made the infinite loop mistake again, should have saved it to ../hq_macaddsearch.txt :/

I grep-ed for "Mac address" in the entire Wireshark directory with grep -ir "mac address" . and found over 500 entries. I would go back to it after reading the docs to understand which entries are important.

Reading Wireshark docs:

• The overview of the Wireshark
  
  • Dumpcap needs elevated privileges to do the capturing. It has code customized for each OS. It talks with WinPcap or libpcap, platform-dependent packet capture libraries to capture from different types of interfaces.
    
• In the different dissectors in epan/dissectors decodes a layer of the protocol before handing it to the next level of dissector.
• Wireshark is based on the GTK toolkit for the UI.
• Not very helpful in trying to find MAC addresses

epan = Ethereal Packet ANalyzer
ett = Ethereal Tree Type
hf = header field
gchar* = pointer to character string
guint8 = 1-byte value
guint16 = 2-byte value
tvb = Testy Virtual Buffer

Reading Wireshark & Ethereal Network Protocol Analyzer Toolkit

• every NIC (Network Interface Card) has a unique MAC Address
• MAC adress is a 48-bit number, aka 12-digit hexadecimal number. The first half identifies the manufacturer of the Ethernet card, last 24 bits is a serial number assigned. e.g.:12:34:56:78:9a:bc
• View MAC Address on different operating systems:
• Windows XP: command line type ipconfig /all, under Physical Address
  
• Linux: command line type ifconfig -a, listed as "HWaddr". "lo" doesn't have a MAC address, because it's a loop back address only. On other computersarp -a might be useful too
  
• MacOS: command line type ifconfig -a, listed as "Ether" label.

Wireshark Source Code to help out

SVN Get the source code with SVN. Code can also be viewed online.

Resources:
- Wireshark Devolpement Guide, especially read this section on Wireshark Developement
- Wireshark Development Wiki
- doc/README.developer
- Wireshark User's Guide

Wireshark uses libpcap (like WinPcap) to do packet capture.

Old to-do for Wireshark (ranked in terms of easiness)
a) find where it communicates to ethernet
--1 read the section on Wireshark Developement
b) MAC addresses
--2 read how the Victor's EtherShell gets Mac address for Linux
--3 get MAC addresses in Windows and MacOS
(4) ... if I run into trouble, set up Wireshark on a vm to test it

Listing All Network Devices on Linux: finding how

A continuation of my previous post. (thanks for the help from hly).

int main(int argc, char **argv)
  char **argv is the same as char * argv[]

To find how to list all network devices, use the ifconfig source code, starting with the "ifconfig.c" file.

Goal: find the code that generates the "Iface" devices (left-most column of below).

$ ifconfig -s
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500 0         0      0      0 0             0      0      0      0 BMU
lo        16436 0      9359      0      0 0          9359      0      0      0 LRU
vmnet1     1500 0         0      0      0 0          1122      0      0      0 BMRU
vmnet8     1500 0         0      0      0 0          1122      0      0      0 BMRU
wlan0      1500 0    275975      0      0 0        223090      0      0      0 BMRU

Inside "ifconfig.c", I searched for "-s" and got

263     else if (!strcmp(*argv, "-s"))
264         ife_short = 1;

So then I searched for "ife_short", and found it inside "if_print()", which:
1. prints out the top row of text.
-. Since we don't have an ifname specified (like "eth0" or "vmnet8"), we can't just simply lookup_interface(ifname) to get the ife and use
-. do_if_fetch(ife) to check that the ife interface/device is still good (it uses if_fetch())
-. if so, print that ife with ife_print_short(), the column we want is printf("%-5.5s ", ptr->name);

103 static int if_print(char *ifname)
 104 {
 105     int res;
 106 
 107     if (ife_short)
 108         printf(_("Iface   MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX     -OVR Flg\n"));
 109 
 110     if (!ifname) {
 111         res = for_all_interfaces(do_if_print, &opt_a);
 112     } else {
 113         struct interface *ife;
 114 
 115         ife = lookup_interface(ifname); //ife means "interface"
 116         res = do_if_fetch(ife);
 117         if (res >= 0)
 118             ife_print(ife);
 119     }
 120     return res;
 121 }

2. We'll instead do "for_all_interfaces()", which gets all the interfaces (see lib/interface.c)

128 int for_all_interfaces(int (*doit) (struct interface *, void *), void *cookie)
129 {
130     struct interface *ife;
131 
132     if (!int_list && (if_readlist() < 0))
133         return -1;
134     for (ife = int_list; ife; ife = ife->next) {
135         int err = doit(ife, cookie);
136         if (err)
137             return err;
138     }
139     return 0;
140 }

Stuck. Victor told me to look at if_readlist(). "It seems that this function is iterating over the return array and feeding each item to a callback."

to-do ask:
1. Is my above resoning correct?
2. If so, how does for_all_interfaces() work? lib/interface.c Look at if_readlist()
3. How do I make this so I can test it? (I don't want it to screw up my ifconfig, test it in the directory (+m ifconfig.c and then ./ doesn't work))
4. is getting interface from IP addresses helpful at all? with the getifaddrs(): from stack overflow question and from IBM.

An Old To-Do

1. Read this ruby gem
2. Read the bsd version of  "sourcecode ifconfig" to figure out how linux list devices!
• CURRENT: still digging in the source
3. Windows/MacOS: Find code in Wireshark that a) communicates to the ethernet, b) gets the MAC addresses in those operating systems
• set up Windows VM
• CURRENT: got the code
4. Read POSIX socket API document
5. read about MSDN (needs to be more specific)

Listing All Network Devices on Linux: ifconfig vs "ip addr" in iproute2

To list all the networking interfaces on Linux, don't use ifconfig because it's deprecated, use ip addr instead. Ip addr is found in a utilities collection called iproute2.  The wiki link points to where to download iproute2 source code, which was last updated on March 17, 2011 as I write this. I also finally found ifconfig's source code (in C) from net-tools. Note that its last update was in 2001, a decade ago!

ifconfig

eth0      Link encap:Ethernet  HWaddr 00:21:70:92:5b:c5
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:17

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:20236 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20236 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3259142 (3.2 MB)  TX bytes:3259142 (3.2 MB)

vmnet1    Link encap:Ethernet  HWaddr 00:50:56:c0:00:01
          inet addr:172.16.76.1  Bcast:172.16.76.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2461 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

vmnet8    Link encap:Ethernet  HWaddr 00:50:56:c0:00:08
          inet addr:172.16.197.1  Bcast:172.16.197.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2461 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan0     Link encap:Ethernet  HWaddr 00:1c:26:64:0f:ac
          inet addr:18.111.100.150  Bcast:18.111.127.255  Mask:255.255.224.0
          inet6 addr: fe80::21c:26ff:fe64:fac/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:191734 errors:0 dropped:0 overruns:0 frame:0
          TX packets:156739 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:224807742 (224.8 MB)  TX bytes:28859347 (28.8 MB)

See, they both give the same list!

ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 00:21:70:92:5b:c5 brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1c:26:64:0f:ac brd ff:ff:ff:ff:ff:ff
    inet 18.111.100.150/19 brd 18.111.127.255 scope global wlan0
    inet6 fe80::21c:26ff:fe64:fac/64 scope link
       valid_lft forever preferred_lft forever
4: vmnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:50:56:c0:00:01 brd ff:ff:ff:ff:ff:ff
    inet 172.16.76.1/24 brd 172.16.76.255 scope global vmnet1
    inet6 fe80::250:56ff:fec0:1/64 scope link
       valid_lft forever preferred_lft forever
5: vmnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:50:56:c0:00:08 brd ff:ff:ff:ff:ff:ff
    inet 172.16.197.1/24 brd 172.16.197.255 scope global vmnet8
    inet6 fe80::250:56ff:fec0:8/64 scope link
       valid_lft forever preferred_lft forever

Net-tools-1.60's "ifconfig.c" file has about 1100 lines of code.
Iproute2-2.6.38/ip/ipaddress.c file has about 1200 lines of code.