hpfriends - real-time social data-sharing

Outline

1. About us
2. Motivation
3. hpfeeds
4. hpfriends
5. Technology

About us

Mark 'rep' Schloesser

• Malware researcher at Rapid7
• Honeynet member since 2008

Jojo 'heipei' Gilger

• PhD student, IT-Security, RWTH Aachen University
• Honeynet member since 2012

ITSec @ RWTH

• Lectures: ITSec 1+2, Mobile Security, Security in Wireless Networks
• Lab: Security Lab (Hands-on practical offensive security)
• CTF: 0ldeur0pe, rwthCTF 2011 & 2012

itsec.rwth-aachen.de

The Honeynet Project

Motivation: Sharing stuff

Motivation: The HoneyMap

Available options for sharing data

The usual suspects...

• Batch: FTP, E-Mail, scp, One-click hoster ...
• Real-time: XMPP, HTTP submission

... with their common problems.

• Naming and formatting of data. Channels?
• Authentication? Access Control?
• Protocol and management overhead?

Honeynet Generic Datafeeds

The hpfeeds protocol

hpfeeds is an authenticated publish/subscribe datafeed protocol (by Mark) used by the Honeynet Project. It supports channels and authkeys.

• Channel permissions on central webif connected to the broker
• Access Control via Authkeys, rather than users
• Simple wire-format, easy to implement and use
• Implementations for hpfeeds exist for Python, Go, Ruby and JavaScript

hpfeeds - details

• Five protocol message types: PUB, SUB, ERROR, INFO, AUTH
• Central feedbroker on server, reference client utility
• Submission modules for Dionaea, Glastopf and Kippo

• Statistics sink
• Webinterface for managing ACLs

hpfeeds - pros & cons

Pros

• + Easy data sharing, more people using it inside HoneyNet project
• + Software support improving steadily
• + Even third parties are interested

Cons

• - Channels are global (only one dionaea.capture channel)
• - Management of access control falls back to central admin (== Mark)

So what would we need?

A system for large-scale real-time data-sharing would have to ...

1. be real-time (duh!)
2. follow simple pub/sub principles
3. allow for distributed and natural access control

Distributed Access Control

Managing permissions for a large number of users does not scale and also does not correspond to the way we collaborate. Think "Web of Trust" and "Distributed Version Control" (like with git).

Social is the solution!

hpfriends

hpfriends is an evolution of the hpfeeds system. The wire-format is identical, the message broker behaves the same way. The big difference is the data-sharing model and the backend software.

hpfriends - unique features

hpfriends supports users and groups. These are connected by sharing relations, representing a web of trust between users who trust each other by sharing their own data as well as data received by third parties.

hpfriends - social graph

hpfriends operates on a social graph. Users and groups are nodes, sharing relations are edges.

Edges between users have a type (shares, shares_all) and an attribute, i.e. the channel name.

Group membership is also stored as an edge between a user and group.

hpfriends - channels

Channels behave quite differently than in the original hpfeeds.

• I can publish on any channel-name I like
• I can add sharing-relationships with groups or people
• Each user can decide whether to accept the share I am offering

hpfriends - demo time!

We're going to show you ...

1. A simple 1:1 data sharing scenario
2. A scenario where data is also relayed to a third party
3. The way the HoneyMap works behind the scenes

hpfriends - implementation

Currently, hpfriends uses the Neo4j graph database to store user, group and sharing information. Further information on users / etc is stored within MongoDB.

For actual message delivery via the feedbroker, sharing relationships are cached for 5-minute intervals for performance.

hpfriends - Neo4j query

start usrc=node({srcid}), udst=node({subids})
        match p=usrc-[:*1..]->udst, udst-[ra:accepts]->umid
        with relationships(p) as rs, udst as udst, ra as ra
        where id(last(rs)) = ra.sid
        return distinct udst'''

hpfriends - frontend

Public hpfriends webif is at hpfriends.honeycloud.net. The webif was built using Meteor, a real-time JavaScript / NodeJS web-framework which takes care of data persistence & synchronization, data binding to the UI, templating, latency compensation and many other things.

• CoffeeScript with Handlebars templating
• MongoDB and Neo4j queries in the background
• Web 2.0 stuff: Twitter Bootstrap, Webfonts, Websockets ...

hpfriends - hpfeeds wire format

struct MsgHeader {
  uint32_t messageLength; // total length
  uint8_t opCode; // currently five opCodes implemented
};

length | opcode | next | identifier | next | channelname | payload
--------------------------------------------------------------------------
    85        3   9      b4aa2@hp1    9      mwcapture     137941a3d8589f9
    

Full documentation and list of opCodes at github/rep/hpfeeds.

hpfriends - Software support

• dionaea (server honeypot): branch which has hpfeeds integration

• kippo (medium-interaction ssh honeypot): rudimentary
• thug (low-interaction honeyclient): hpfeeds integrated

Your software as well?

hpfriends - links

• hpfriends at hpfriends.honeycloud.net
  • Free to use, source not open (yet)
  • Consider contributing bug-reports / feature ideas!
• Guide on how to use hpfriends at heipei.github.io
• hpfeeds repo and documentation at github.com/rep/hpfeeds
• Thanks to Florian Weingarten for Ruby / Go implementation
• The HoneyMap at map.honeynet.org

Thank you for listening!

Follow-up questions to
Mark - @repmovsb
Jojo - @heipei