In the world of security, it is important to understand how various tools exchange information. This knowledge helps in extending the solutions or creating valuable add-ons. This post describes some formats developed to represent data collected by security monitoring systems at the network level. Many of them have not been formally standardized and further analysis is generally required to extract useful (i.e., actionable) information.
- NetFlow. NetFlow is a protocol developed by Cisco[RFC 3954] in early 1990s and was originally developed for exporting traffic summaries in the form of IP flow records from active network devices (i.e., routers, switches) that is now used by many passive flow sensors. While it was introduced by Cisco Systems, similar export features are present in networking equipment from other vendors, for example, Jflow [Juniper Networks], NetStream [HP], Cflowd [Alcatel-Lucent], Rflow [Ericsson], AppFlow [Citrix], Traffic Flow [MikroTik] and sFlow [multi-vendor]. Netflow data is produced by a network device or sensor which transfers the data by either UDP or SCTP to a collector that aggregates and organizes the data so that it can be queried and analyzed. NetFlow v5 used a fixed record format which was changed in NetFlow v9 to customized record formats.
- IPFIX. Internet Protocol Flow Information Export [IPFIX] was published [RFC 3917] in 2004 by IETF. It has evolved from NetFlow that is formalized in an RFC that defines how the flow information should be formatted for export from network devices and sensors (“flow meters”). Like NetFlow records, IPFIX records describe a single logical IP connection corresponding to the 5-tuple in an IP header, generally include fields describing traffic volumes (in bytes and packets) for the connection, but may also include any number of other fields that summarize information about the connection.
- PCAP. Packet Capture File was published in 1998 (libpcap 0.4) by the TCPDUMP project. PCAP is the format used by many popular packet capture tools, and is used to store or transmit captured network traffic. The format is very simple and allows the storage of time zone, clock accuracy and link type along with the captured network packets.
- PcapNG. PCAP Next Generation Dump File Format was published in 2004 by the Wireshark project. PcapNg is a format for the storage and transmission of packet traces. The primary design goals of PcapNg are extensibility and portability. The format allows additional descriptive data to be attached to PCAP traces. The PcapNg standard defines a format for representing additional capture metadata. This includes facilities for describing the capture device and the filter used at capture time. It also includes provisions for storing NetFlow and Remote Network Monitoring (RMON) data.
- CEF. Common Event Format is a syslog-based format from HP, for the transmission of event information between event producers and consumers. CEF messages contain data about the originating device, event signature, a human readable description, and severity. The specification also includes a dictionary of predefined keys for describing security-related events. This format was made famous by the ArcSight SIEM tool.
- Syslog. Syslog was developed in the 1980s by Eric Allman as part of the Sendmail project, and was initially used solely for Sendmail. Syslog has since become the standard logging solution on Unix and Unix-like systems; there have also been a variety of syslog implementations on other operating systems and is commonly found in network devices such as routers. The Internet Engineering Task Force documented the status quo in RFC 3164 and later standardized by RFC 5424.
- Microsoft Event Log Format. This is the proprietary Microsoft format used to log all events for Windows desktops and servers. Since the Windows desktops form a large percentage of the enterprise environment, this format becomes important to be able to extract any security related information from the system.
These are some of the formats used by many of the security tools. Understanding these in some detail help us to create solutions which can automate & integrate multiple tools.
Looking forward for comments / suggestion. Please add to the list.