@sunny-r I understand your thought. What I can say is that:

• 1 We did a big part of it as when we started NFStream it was mainly due to the lack of reliability of existing measurement tools (not the same results, etc.) so we spent a lot of time checking the results and comparing pcaps to flows manually. As there no bug-free software, I understand that you need that but you can reduce it to one random file check from time to time.
• 2 Within NFStream internal structures, there is all nDPI structures exposed. So, do not hesitate to add your fields to NFStream. It's as simple as a copy from a struct field to a Python object.
• 3 And you are right! :D

Ok, I don't doubt you guys have been careful, but picking through a few examples also helps me understand what's going on, but like you said, it's only the odd random check.

I'd like to have a go at 2) - using built-in nDPI to extract fields of interest. Can you give me an example to help me get started, I've not done this before. Say we have a DNS response with two Answers as CNAMEs, how would the hosts and IPs be exposed? I'm not really sure how to go from ndpi/structs to python objects.

Thanks!

Hi,

I'm struggling to expose the pre-extracted nDPI fields in a flow; I've started writing an NFPlugin, and found an object that I've tried to pull out for some analysis:

    def on_expire(self, flow):
        flow.udps.ndpi_flow = flow._C.ndpi_flow

And this is the error I see:

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _cffi_backend._CDataBase objects

Can I get some help on how to extract fields already identified by the nDPI engine?

Thanks

Sunny

bump; @aouinizied - how can i expose more of the ndpi fields that are extracted? Do I need to use NFPlugin?

@sunny-r It will depends on your needs. There is mainly two ways to do it.

• The first one is to extend the source code of NFStream to handle additional fields from nDPI.
• Using NFPlugin as you started.

Let's focus on 2 as it is the simplest way (less efficient however).

As you can see ndpi_flow field is under protected attribute of NFlow and this is because user are not supposed to play with it (specially overwriting it).
Now if you want to read it, this can be achieved, however, you must keep in mind some points about how NFStream works internally and some general knowledge about how C structures are handled within Python:

• Whatever you wan to create as a plugin field, it must be a Python object. This is due to the fact that NFStream several meters processes that communicate with the parent process using sockets and thus, everything is pickled at some point. This explain why you have this error, you assign to the attribute ndpi_flow structure which is a C structure and pickle don't know how to serialize it.
• You must keep in mind that internally (for memory saving purposes), ndpi references (flow_struct_src_id_struct and dst_id struct) are freed as soon as the detection is completed. Consequently, if you access this structure when it is freed, you will have issues.
• So, you can access it mainly on_init and on_update before detection is completed. Note that with such approach you will not be called on the packet when the detection is considered as completed (internal logic of NFStream that you can check in engine.c)

Please find in the following a working example on how to extract host_server_name for nDPI flow structure within a plugin.

from nfstream import NFStreamer, NFPlugin
import cffi

class NDPIExtractor(NFPlugin):
    def on_init(self, packet, flow):
        flow.udps.my_field = None
        if flow._C.detection_completed == 0:  # This avoid attempt to read a freed ndpi_flow structure
            flow.udps.my_field = self.ffi.string(
                flow._C.ndpi_flow.protos.tls_quic_stun.tls_quic.client_requested_server_name
            ).decode('utf-8')

    def on_update(self, packet, flow):
        if flow._C.detection_completed == 0:  # This avoid attempt to read a freed ndpi_flow structure
            flow.udps.my_field = self.ffi.string(
                flow._C.ndpi_flow.protos.tls_quic_stun.tls_quic.client_requested_server_name
            ).decode('utf-8')

    # Not using on_expire to avoid manipulating ndpi_flow as it is already freed


flow_streamer = NFStreamer(source="tests/facebook.pcap",  udps=NDPIExtractor(ffi=cffi.FFI()))
for flow in flow_streamer:
   print(flow)

As in this example the targeted field is a char [fixed_size], some casting through cffi is needed to convert it to a string. This is not needed if you will work on integer fields of nDPI structures.

Hope this helps!

Zied