[Dataset] mit/reality (v. 2005-07-01)

top

the initial version

@MISC{mit-reality-2005-07-01,
  author = {Nathan Eagle and Alex (Sandy) Pentland},
  title = {{CRAWDAD} data set mit/reality (v. 2005-07-01)}, 
  howpublished = {Downloaded from http://crawdad.cs.dartmouth.edu/mit/reality},
  month = jul,  
  year = 2005
}
					

The authors have captured communication, proximity, location, and activity information from 100 subjects at MIT over the course of the 2004-2005 academic year.  This data represents over 350,000 hours (~40 years) of continuous data on human behavior.  Such rich data on complex social systems have implications for a variety of fields.

Our study consists of one hundred Nokia 6600 smart phones pre-installed 
with several pieces of software we have developed as well as a version of 
the Context application from the University of Helsinki. 
Seventy-five users are either students or faculty in the MIT Media Laboratory, 
while the remaining twenty-five are incoming students at the MIT Sloan business 
school adjacent to the laboratory. Of the seventy-five users at the lab, 
twenty are incoming masters students and five are incoming MIT freshman.

We exploit the fact that modern phones use both a short-range RF network 
(e.g., Bluetooth) and a long-range RF network (e.g., GSM), and that 
the two networks can augment each other for location and activity inference.
We logged cell tower ID to determine approximate location and at the same
time we logged  Bluetooth devices. 
Bluetooth is a wireless protocol in the 2.40-2.48 GHz range, developed 
by Ericsson in 1994 and released in 1998 as a serial-cable replacement 
to connect different devices.

The information we are collecting includes call logs, Bluetooth devices in proximity, 
cell tower IDs, application usage, and phone status (such as charging and idle), 
which comes primarily from the Context application. The study will generate 
data collected by one hundred human subjects over the course of nine months and 
represent approximately 500,000 hours of data on users' location, communication 
and device usage behavior.

[Traceset] mit/reality/blueaware (v. 2005-07-01)

top

the initial version

@MISC{mit-reality-blueaware-2005-07-01,
  author = {Nathan Eagle and Alex (Sandy) Pentland},
  title = {{CRAWDAD} trace set mit/reality/blueaware (v. 2005-07-01)}, 
  howpublished = {Downloaded from http://crawdad.cs.dartmouth.edu/mit/reality/blueaware},
  month = jul,  
  year = 2005
}
					

The authors have captured communication, proximity, location, and activity information from 100 subjects at MIT over the course of the 2004-2005 academic year.  This data represents over 350,000 hours (~40 years) of continuous data on human behavior.

Every Bluetooth device is capable of device-discovery, which allows them 
to collect information on other Bluetooth devices within 5-10 meters. 
This information includes the Bluetooth MAC address (BTID), device name, and 
device type. The BTID is a hex number unique to the particular device. 
The device name can be set at the user's discretion; 
e.g., Tony's Nokia. Finally, the device type is a set of three integers 
that correspond to the device discovered; e.g., Nokia mobile phone, or IBM laptop.

To log BTIDs we designed a software application, BlueAware, that runs passively
in the background on MIDP2-enabled mobile phones. Bluetooth was primarily designed
to enable wireless headsets or laptops to connect to phones, but as a byproduct,
devices are becoming aware of other Bluetooth devices carried by people
nearby. Our application records and timestamps the BTIDs encountered in a proximity
log and makes them available to other applications. BlueAware is automatically 
run in the background when the phone is turned on, making it essentially invisible 
to the user.

Bluedar was developed to be placed in a social setting and continuously scan 
for visible devices, wirelessly transmitting detected BTIDs to a server 
over an 802.11b network. The heart of the device is a Bluetooth
beacon designed by Mat Laibowitz incorporating a class 2 Bluetooth chipset that can
be controlled by an XPort web server. We integrated this beacon with an
802.11b wireless bridge and packaged them in an unobtrusive box. An application
was written to continuously telnet into multiple BlueDar systems, repeatedly scan for
Bluetooth devices, and transmit the discovered proximate BTIDs to our server. Because
the Bluetooth chipset is a class 2 device, it is able to detect any visible Bluetooth
device within a working range of up to twenty-five meters.

1. All the data from a phone are stored on a flash memory card, which has a finite 
number of read-write cycles. Initial versions of our application wrote over the same 
cells of the memory card. This led to failure of a new card after about a month of data 
collection, resulting in the complete loss of data. When the application was changed 
to store the incremental logs in RAM and subsequently write each complete log 
to the flash memory, our data corruption issues virtually vanished. 
However, ten cards were lost before this problem was identified, destroying portions 
of the data collected during the months of September and October for six Sloan 
students and four Media Lab students.

2. Another source of missing data is due to powered-off devices. 
On average we have logs accounting for approximately 85.3% of the time 
since the phones have been deployed. Less than 5% of this is due to data corruption, 
while the majority of the missing 14.7% is due to almost one fifth of the subjects 
turning off their phones at night.

3. There is a small probability (between 1-3% depending on the phone) 
that a proximate, visible device will not be discovered during a scan. 
Typically this is due to either a low level Symbian crash of an application 
called the "BTServer", or a lapse in the device discovery protocol. The BT server 
crashes and restarts approximately once every three days 
(at a 5 minute scanning interval) and accounts for a small fraction of the total error. 
However, to detect other subjects, we can leverage the redundancy implicit in the system. 
Because both of the subjects' phones are actually scanning, the probability of 
a simultaneous crash or device discovery error is less than 1 in 1000 scans.

1. Continually scanning and logging BTIDs can expend an older mobile phone battery 
in about 18 hours.  While continuous scans provide a rich depiction of a user's 
dynamic environment, most individuals expect phones to have standby times exceeding 
48 hours. Therefore BlueAware was modified to only scan the environment once every 
five minutes, providing at least 36 hours of standby time.

2. While the custom logging application on the phone crashes occasionally 
(approximately once every week), these crashes fortunately do not result 
in significant data loss. An additional small application was written to start 
on boot and continually review the running processes on the phone, 
verifying that our logging application is always running. Should there be a time 
where this is not the case, the application is immediately restarted. 
This functionality also ensures that logging begins immediately once the phone 
is turned on. However, while this logging application is now fairly robust 
and can be assumed to be running anytime the phone is on, the dataset generated is 
certainly not without noise. 

3. By scanning only periodically every five minutes, shorter proximity events 
may be missed.

1. The ten meter range of Bluetooth along with the fact that it can penetrate 
some types of walls, means that people not physically proximate may incorrectly 
be logged as such. 

2. An error comes from the phone being either explicitly turned off by the user 
or exhausting the batteries. According to our collected survey data, users report 
exhausting the batteries approximately 2.5 times each month. One fifth of our subjects 
manually turn the phone off on a regular basis during specific contexts such 
as classes, movies, and (most frequently) when sleeping. Immediately before the phone 
powers down, the event is timestamped and the most recent log is closed. A new log 
is created when the phone is restarted and again a timestamp is associated with the event.

				3. A more critical source of error occurs when the phone is left on, but not carried 
by the user. From surveys, we have found that 30% of our subjects claim to never 
forget their phones, while 40% report forgetting it about once each month, and 
the remaining 30% state that they forget the phone approximately once each week. 
Identifying the times where the phone is on, but left at home or in the office presents 
a significant challenge when working with the dataset. To grapple with the problem, 
we have created a 'forgotten phone' classifier. Features included staying in the same 
location for an extended period of time, charging, and remaining idle through missed 
phone calls, text messages and alarms. When applied to a subsection of the dataset 
which had corresponding diary text labels, the classifier was able to identify 
the day where the phone was forgotten, but also mislabeled a day when the user 
stayed home sick. By ignoring both days, we risk throwing out data on outlying days, 
but have greater certainty that the phone is actually with the user. A significantly 
harder problem is to determine whether the user has temporarily moved beyond ten meters 
of his or her office without taking the phone. Empirically, this appears to happen 
with many subjects on a regular basis and there doesn't seem to be enough unique 
features of the event to accurately classify it. However, this phenomenon does not 
diminish the extremely strong correlation between detected proximity and self-report 
interactions. Lastly, while frequency of proximity within the workplace can be useful, 
the most salient data comes from detecting a proximity event outside MIT, 
where temporarily forgetting the phone is less likely to repeatedly occur.

In return for the use of the Nokia 6600 phones, students have been asked to 
fill out web-based surveys regarding their social activities and the people 
they interact with throughout the day. Comparison of the logs with survey data 
has given us insight into our dataset's ability to accurately map social network 
dynamics. Through surveys of approximately forty senior students, we have validated 
that the reported frequency of (self-report) interaction is strongly correlated 
with the number of logged BTIDs (R=.78, p=.003), and that the dyadic self-report 
data has a similar correlation with the dyadic proximity data (R=.74, p~=.0001). 
Additionally, a subset of subjects kept detailed activity diaries over several months. 
Comparisons revealed no systematic errors with respect to proximity and location, 
except for omissions due to the phone being turned off.

[Trace] mit/reality/blueaware/activityscpan (v. 2005-07-01)

top

The initial version

@MISC{mit-reality-blueaware-activityscpan-2005-07-01,
  author = {Nathan Eagle and Alex (Sandy) Pentland},
  title = {{CRAWDAD} trace mit/reality/blueaware/activityscpan (v. 2005-07-01)}, 
  howpublished = {Downloaded from http://crawdad.cs.dartmouth.edu/mit/reality/blueaware/activityscpan},
  month = jul,  
  year = 2005
}
					

Activity span logs

oid, endtime, starttime, person_oid

activity span logs

[Trace] mit/reality/blueaware/callspan (v. 2005-07-01)

top

The initial version

@MISC{mit-reality-blueaware-callspan-2005-07-01,
  author = {Nathan Eagle and Alex (Sandy) Pentland},
  title = {{CRAWDAD} trace mit/reality/blueaware/callspan (v. 2005-07-01)}, 
  howpublished = {Downloaded from http://crawdad.cs.dartmouth.edu/mit/reality/blueaware/callspan},
  month = jul,  
  year = 2005
}
					

Call span logs

oid, endtime, starttime, person_oid, phonenumber_oid, callid, contact, description, direction, duration, number, status, remote

"person_oid" refers to the person running the software on their phone,
for which this call was logged. It is who this callspan is 'attached'
to, and will always be attached to some person_oid.

"direction" refers to the direction of the call from the perspective of
this particular person/cellphone that recorded this callspan (the same
as the person referred to by person_oid). Can be Incoming, Missed Call,
or Outgoing.

"phonenumber_oid" refers to the number 'on the other end' of the
network, which may be a landline, a cell phone line, or even that phone
network's voicemail.

So in other words, person_oid and phonenumber_oid represent the two ends
of the phone call, with the direction of the phone call represented in
the direction field. If you want to utilize all 897921 callspan records,
you might want to define these "calls" as between two phonenumbers,
instead of as between two persons. So the call would exist between
callspan.person_oid's phonenumber_oid, and the callspan.phonenumber_oid.


In addition, if the callspan records a call between two people that were
running the software and part of the study (they both are part of the
study), then there are a few additional properties that will hold about
the callspan:

For some person src: src.oid = callspan.person_oid (for all calls)
For some person dst: dst.phonenumber_oid = callspan.phonenumber_oid
(only for in-network calls)

There should also be a symmetric callspan going in the other direction.
For some callspan Y:
  Y.person_oid == dst.oid
  Y.phonenumber_oid = src.phonenumber_oid

call span logs

[Trace] mit/reality/blueaware/cellspan (v. 2005-07-01)

top

The initial version

@MISC{mit-reality-blueaware-cellspan-2005-07-01,
  author = {Nathan Eagle and Alex (Sandy) Pentland},
  title = {{CRAWDAD} trace mit/reality/blueaware/cellspan (v. 2005-07-01)}, 
  howpublished = {Downloaded from http://crawdad.cs.dartmouth.edu/mit/reality/blueaware/cellspan},
  month = jul,  
  year = 2005
}
					

Cell span logs

oid, endtime, starttime, person_oid, celltower_oid

cell span logs

[Trace] mit/reality/blueaware/coverspan (v. 2005-07-01)

top

The initial version

@MISC{mit-reality-blueaware-coverspan-2005-07-01,
  author = {Nathan Eagle and Alex (Sandy) Pentland},
  title = {{CRAWDAD} trace mit/reality/blueaware/coverspan (v. 2005-07-01)}, 
  howpublished = {Downloaded from http://crawdad.cs.dartmouth.edu/mit/reality/blueaware/coverspan},
  month = jul,  
  year = 2005
}
					

Cover span logs

oid, endtime, starttime, person_oid

cover span logs

[Trace] mit/reality/blueaware/devicespan (v. 2005-07-01)

top

The initial version

@MISC{mit-reality-blueaware-devicespan-2005-07-01,
  author = {Nathan Eagle and Alex (Sandy) Pentland},
  title = {{CRAWDAD} trace mit/reality/blueaware/devicespan (v. 2005-07-01)}, 
  howpublished = {Downloaded from http://crawdad.cs.dartmouth.edu/mit/reality/blueaware/devicespan},
  month = jul,  
  year = 2005
}
					

Device span logs

oid, endtime, starttime, person_oid, device_oid

device span logs

[Author] Nathan Eagle

top

[Author] Alex (Sandy) Pentland

top

[Paper] carreras-malware

top

[Paper] daly-social

top

Message delivery in sparse Mobile Ad hoc Networks (MANETs) is difficult due to 
the fact that the network graph is rarely (if ever) connected. A key challenge 
is to find a route that can provide good delivery performance and low 
end-to-end delay in a disconnected network graph where nodes may move freely. 
This paper presents a multidisciplinary solution based on the consideration of 
the socalled small world dynamics which have been proposed for economy and 
social studies and have recently revealed to be a successful approach to be 
exploited for characterising information propagation in wireless networks. To 
this purpose, some bridge nodes are identified based on their centrality 
characteristics, i.e., on their capability to broker information exchange among 
otherwise disconnected nodes. Due to the complexity of the centrality metrics 
in populated networks the concept of ego networks is exploited where nodes are 
not required to exchange information about the entire network topology, but 
only locally available information is considered. Then SimBet Routing is 
proposed which exploits the exchange of pre-estimated ‘betweenness’ 
centrality metrics and locally determined social ‘similarity’ to the 
destination node. We present simulations using real trace data to demonstrate 
that SimBet Routing results in delivery performance close to Epidemic Routing 
but with significantly reduced overhead. Additionally, we show that Sim- Bet 
Routing outperforms PRoPHET Routing, particularly when the sending and 
receiving nodes have low connectivity.

[Paper] eagle-mobile-phones

top

[Paper] eagle-reality

top

We introduce a system for sensing complex social systems with data collected 
from one hundred mobile phones over the course of six months. We demonstrate 
the ability to use standard Bluetooth-enabled mobile telephones to measure 
information access and use in different contexts, recognize social patterns in 
daily user activity, infer relationships, identify socially significant 
locations, and model organizational rhythms.

[Paper] hui-bubble

top

In this paper we seek to improve understanding of the structure of human 
mobility, and to use this in the design of forwarding algorithms for Delay 
Tolerant Networks for the dissemination of data amongst mobile users. 
Cooperation binds but also divides human society into communities. Members of 
the same community interact with each other preferentially. There is structure 
in human society. Within society and its communities, individuals have varying 
popularity. Some people are more popular and interact with more people than 
others; we may call them hubs. Popularity ranking is one facet of the 
population. In many physical networks, some nodes are more highly connected to 
each other than to the rest of the network. The set of such nodes are usually 
called clusters, communities, cohesive groups or modules. There is structure to 
social networking. Different metrics can be used such as information flow, 
Freeman betweenness, closeness and inference power, but for all of them, each 
node in the network can be assigned a global centrality value. What can be 
inferred about individual popularity, and the structure of human society from 
measurements within a network? How can the local and global characteristics of 
the network be used practically for information dissemination? We present and 
evaluate a sequence of designs for forwarding algorithms for Pocket Switched 
Networks, culminating in Bubble, which exploit increasing levels of information 
about mobility and interaction.

[Paper] hui-community

top

Community is an important attribute of Pocket Switched Networks (PSN), because 
mobile devices are carried by people who tend to belong to communities. We 
analysed community structure from mobility traces and used for forwarding 
algorithms [12], which shows significant impact of community. Here, we propose 
and evaluate three novel distributed community detection approaches with great 
potential to detect both static and temporal communities. We find that with 
suitable configuration of the threshold values, the distributed community 
detection can approximate their corresponding centralised methods up to 90% 
accuracy.

[Paper] karagiannis-power-law

top

We examine the fundamental properties that determine the basic performance 
metrics for opportunistic communications. We first consider the distribution of 
inter-contact times between mobile devices. Using a diverse set of measured 
mobility traces, we find as an invariant property that there is a 
characteristic time, order of half a day, beyond which the distribution decays 
exponentially. Up to this value, the distribution in many cases follows a power 
law, as shown in recent work. This power law finding was previously used to 
support the hypothesis that inter-contact time has a power law tail, and that 
common mobility models are not adequate. However, we observe that the time 
scale of interest for opportunistic forwarding may be of the same order as the 
characteristic time, and thus the exponential tail is important. We further 
show that already simple models such as random walk and random waypoint can 
exhibit the same dichotomy in the distribution of inter-contact time asc in 
empirical traces. Finally, we perform an extensive analysis of several 
properties of human mobility patterns across several dimensions, and we present 
empirical evidence that the return time of a mobile device to its favorite 
location site may already explain the observed dichotomy. Our findings suggest 
that existing results on the performance of forwarding schemes based on 
power-law tails might be overly pessimistic.

[Paper] pitkanen-redundancy

top

[Paper] yoneki-community-detection

top

Community is an important attribute of Pocket Switched Networks (PSNs), since 
mobile devices are carried by people who tend to belong to communities in their 
social life. We discover the heterogeneity of human interactions such as 
community formation from real world human mobility traces. We have introduced 
novel distributed community detection approaches and evaluated with those 
traces. This paper describes a series of visualizations to show characteristics 
of human mobility traces including community detection. We focus on extracting 
information related to levels of clustering, network transitivity, and strong 
community structure. The progression of the connection map along the community 
formation process is also visualized.