OpenEDGAR: Open Source Software for SEC EDGAR Analysis (Michael Bommarito, Daniel Martin Katz & Eric Detterman)

Our next paper — OpenEDGAR – Open Source Software for SEC Edgar Analysis is now available.  This paper explores a range of #OpenSource tools we have developed to explore the EDGAR system operated by the US Securities and Exchange Commission (SEC).  While a range of more sophisticated extraction and clause classification protocols can be developed leveraging LexNLP and other open and closed source tools, we provide some very simple code examples as an illustrative starting point.

Click here for Paper:   < SSRN > < arXiv >
Access Codebase Here: < Github >

OpenEDGAR is an open source Python framework designed to rapidly construct research databases based on the Electronic Data Gathering, Analysis, and Retrieval (EDGAR) system operated by the US Securities and Exchange Commission (SEC). OpenEDGAR is built on the Django application framework, supports distributed compute across one or more servers, and includes functionality to (i) retrieve and parse index and filing data from EDGAR, (ii) build tables for key metadata like form type and filer, (iii) retrieve, parse, and update CIK to ticker and industry mappings, (iv) extract content and metadata from filing documents, and (v) search filing document contents. OpenEDGAR is designed for use in both academic research and industrial applications, and is distributed under MIT License at

LexNLP: Natural Language Processing and Information Extraction For Legal and Regulatory Texts (Bommarito, Katz, Detterman)

Paper Abstract – LexNLP is an open source Python package focused on natural language processing and machine learning for legal and regulatory text. The package includes functionality to (i) segment documents, (ii) identify key text such as titles and section headings, (iii) extract over eighteen types of structured information like distances and dates, (iv) extract named entities such as companies and geopolitical entities, (v) transform text into features for model training, and (vi) build unsupervised and supervised models such as word embedding or tagging models. LexNLP includes pre-trained models based on thousands of unit tests drawn from real documents available from the SEC EDGAR database as well as various judicial and regulatory proceedings. LexNLP is designed for use in both academic research and industrial applications, and is distributed at

Slides from our Presentation at UPenn Computational Linguistics (CLUNCH) / Linguistic Data Consortium (LDC)

We have spent the past couple days at the University of Pennsylvania where we presented information about our efforts to compile a complete United States Supreme Court Corpus.  As noted in the slides below, we are interested in creating a corpus containing not only every SCOTUS opinion, but also every SCOTUS disposition from 1791-2010. Slight variants of the slides below were presented at the Penn Computational Linguistics Lunch (CLunch) and the Linguistic Data Consortium(LDC).  We really appreciated the feedback and are looking forward to continue our work with the LDC.  For those who might be interested, take a look at the slides embedded below or click on this link:

GraphMovie: A Library for Generating Movies from Dynamic Graphs with igraph

Over the past few months, we’ve developed a library for simply generating dynamic network animations. We’ve used this library in visualizations like (1) Visualizing the Gawaher Interactions of Umar Farouk Abdulmutallab, the Christmas Day Bomber and (2) Dynamic Animation of the East Anglia Climate Research Unit Email Network.  Prior to these visualizations, we’ve used Sonia to produce animations like this one. While certainly a useful program for those without programming expertise, Sonia suffers from a number of issues that make it unusable for large graphs or graphs with many “slices.”  Furthermore, in our experience rendering various movies a number of platform issues with the Quicktime and Flash rendering engines have arisen.  Fixing these problems is possible, but Sonia’s large Java codebase makes for a steep learning curve.  As a result, we’ve decided to release this GraphMovie class so that others can use or possibly improve this library.

In order to use the GraphMovie, you’ll need the following:

  • python (tested with 2.6)
  • igraph for network manipulation and visualization
  • Python Imaging Library for manipulating the image frames
  • mencoder from the MPlayer package for encoding the image frames into a movie

Here are the files, hosted on github:

GraphMovie: Example 1 from Computational Legal Studies on Vimeo.

GraphMovie: Example 2 from Computational Legal Studies on Vimeo.

Programming Dynamic Models in Python-Part 3: Outbreak on a Network

In this post, we will continue building on the basic models we discussed in the first and second tutorials. If you haven’t had a chance to take a look at them yet, definitely go back and at least skim them, since the ideas and code there form the backbone of what we’ll be doing here.

In this tutorial, we will build a model that can simulate outbreaks of disease on a small-world network (although the code can support arbitrary networks).  This tutorial represents a shift away from both:

a) the mass-action mixing of the first two and and

b) the assumption of social homogeneity across individuals that allowed us to take some shortcuts to simplify model code and speed execution. Put another way, we’re moving more in the direction of individual-based modeling.

When we’re done, your model should be producing plots that look like this:

Outbreak on a small-world network
Outbreak on a small-world network

Red nodes are individuals who have been infected before the end of the run, blue nodes are never-infected individuals and green ones are the index cases who are infectious at the beginning of the run.

And your model will be putting out interesting and unpredictable results such as these:

Time vs. # of cases
Time vs. # of cases

In order to do this one, though, you’re going to need to download and install have igraph for Python on your system.

Individual-Based Networks

It is important to make the subtle distinction between individual and agent based models very clear here. Although the terms  are often used interchangeably, referring to our nodes, who have no agency, per se, but are instead fairly static receivers and diffusers of infection, as agents, seems like overreaching. Were they to exhibit some kind of adaptive behavior, i.e., avoiding infectious agents or removing themselves from the population during the infective period, they then become more agent-like.

This is not to under- or over-emphasize the importance or utility of either approach, but just to keep the distinction in mind to avoid the “when all you have is a hammer, everything looks like a nail” problem.

In short, adaptive agents are great, but they’re overkill if you don’t need them for your specific problem.

Small World Networks

The guiding idea behind small-world networks is that they capture some of the structure seen in more realistic contact networks: most contacts are regular in the sense that they are fairly predicable, but there are some contacts that span tightly clustered social groups and bring them together.

In the basic small-world model, an individual is connected to some (small, typically <=8) number of his or her immediate neighbors. Some fraction of these network connections are then randomly re-wired, so that some individuals who were previously distant in network terms – i.e., connected by a large number of jumps – are now adjacent to each other. This also has the effect of shortening the distance between their neighbors and individuals on the other side of the graph. Another way of putting this is that we have shortened the average path length and increased the average reachability of all nodes.

These random connections are sometimes referred to as “weak ties”, as there are fewer of these ties that bridge clusters than there are within clusters. When these networks are considered from a sociological perspective, we often expect to find that the relationship represented by a weak tie is one in which the actors on either end have less in common with each other than they do with their ‘closer’ network neighbors.

Random networks also have the property of having short average path lengths, but they lack the clustering that gives the small-world model that pleasant smell of quasi-realism that makes them an interesting but largely tractable, testing ground for theories about the impact of social structure on dynamic processes.

Installation and Implementation Issues

If you have all the pre-requisites installed on your system, you should be able to just copy and paste this code into a new file and run it with your friendly, local Python interpreter. When you run the model, you should first see a plot of the network, and when you close this, you should see a plot of the number of infections as a function of time shortly thereafter.

Aside from the addition of the network, the major conceptual difference is that the model operates on discrete individuals instead of a homogeneous population of agents. In this case, the only heterogeneity is in the number and identity of each individual’s contacts, but there’s no reason we can’t (and many do) incorporate more heterogeneity (biological, etc.) into a very similar model framework.

With Python, this change in orientation to homogeneous nodes to discrete individuals seems almost trivial, but in other languages it can be somewhat painful. For instance, in C/++, a similar implementation would involve defining a struct with fields for recovery time and individual ID, and defining a custom comparison operator for these structs. Although this is admittedly not a super-high bar to pass, it adds enough complexity that it can scare off novices and frustrate more experienced modelers.

Perhaps more importantly, it often has the effect of convincing programmers that a more heavily object-oriented approach is the way to go, so that each individual is a discrete object. When our individuals are as inert as they are in this model, this ends up being a waste of resources and makes for significantly more cluttered code. The end result can often be a model written in a language that is ostensibly faster than Python, such as C++ or Java, that runs slower than a saner (and more readable) Python implementation.

For those of you who are playing along at home, here are some things to think about and try with this model:

  1. Change the kind of network topology the model uses (you can find all of the different networks available in igraph here).
  2. Incorporate another level of agent heterogeneity: Allow agents to have differing levels of infectivity (Easier); Give agents different recovery time distributions (Harder, but not super difficult).
  3. Make two network models – you can think of them as separate towns – and allow them to weakly influence each other’s outbreaks. (Try to use the object-oriented framework here with minimal changes to the basic model.)

That’s it for tutorial #3, (other than reviewing the comment code which is below) but definitely check back for more on network models!

In future posts, we’ll be thinking about more dynamic networks (i.e., ones where the links can change over time), agents with a little more agency, and tools for generating dynamic visualizations (i.e., movies!) of stochastic processes on networks.

That really covers the bulk of the major conceptual issues. Now let’s work through the implementation.

Click Below to Review the Implementation and Commented Code!

Continue reading “Programming Dynamic Models in Python-Part 3: Outbreak on a Network”

Visualizing Dynamic Networks with Python, Igraph, and SONIA

igraph2sonia Example 1 from michael bommarito on Vimeo.

When it comes to quickly motivating a point or engaging students in a classroom, one of the most effective tools is visualization. Not only do movies provide fun and excitement, but they also allow viewers to leverage the abilities of the visual cortex to infer dynamics and patterns in the animated system.

For our recent research, dynamic graphs are the type of system of interest. As I’ve covered before, Python is my language of choice for most programming tasks. Furthermore, Python is a very accessible language, even for beginners. However, when it comes to visualizing dynamic networks, we need another tool.  Our tool of choice is SONIA, the Social Network Image Animator.

I thought I’d provide a helpful little function to generate SONIA input files from igraph objects, along with a few examples.

This function takes as input an igraph.Graph object and a file name to store the SONIA output in. Every vertex in the Graph object should have a time attributed specified, either simply as an integer indicating the start time, or as a tuple or list of the form (startTime,endTime). Check out the following two examples if you need more guidance. Both examples visualize the construction of a periodic lattice. However, in the second example, nodes decay after some random time. Make sure not to miss the second video at the bottom of the post!

igraph2sonia Example 2 from Michael J Bommarito II on Vimeo.

The Revolution Will Not Be Televised — But Will it Come from HLS or YLS ? A Social Network Analysis of the Legal Academy (Part IV)

Law Prof Diffusion

This is the final installment of posts related to Reproduction of Hierarchy? A Social Network Analysis of the American Law Professoriate. Thanks for your emails.

Here is the plot we provide within the paper.  As a general proposition, we believe this represents an upper bound measure for the intellectual reach of an agenda offered by a given institution.  With respect to our version of the Reed Frost Epidemiological Model, we use the p parameter to model “idea infectiousness.”  When p = 1 every institution “contacted” by the idea is infected with the idea. When p = 0 no institution “contacted” by the idea is infected.  In this version, we use the programming language python to run the model 500 times per institution. The above plot represents an estimate of the “diffusion curve” for each of the 184 institutions in our model. Building off central limit type properties, this leaves a far better estimate of reach than is offered in the single model run from the previous Netlogo GUI.

A cursory review of the above plot demonstrates, we are far from the land of linearity.  Namely, a large number of institutions are able to reach much of the graph with very small changes in the value of p.

In the Structure of Scientific Revolutions, Kuhn quotes from Max Planck:  “a new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.” Following Planck, we believe retirement is indeed be an important mechanism.  However, we also argue the nature of the p parameter is a relevant consideration.  In fact, unpacking various dimensions of p is the key to the broader model. Specifically, what are the properties of an idea that generate its infectiousness? Of course, we might like to believe infectiousness is related to a class of normatively attractive properties such as promoting efficiency or justice.  However, it is not clear that this follows.

We took no pass on the question of whether some institutions would be better or worse at producing ideas with greater or lesser values of p. The motivated question for this post considers whether, in general, the institutions which are top producers of law professors are (1) leaders in innovation, (2) subsequent ratifiers of a newly established paradigm or (3) defenders of the status quo. In a deep sense, we are asking how to reasonably model decision making by the heterogeneous agents located at such institutions.  Do institutions reward or punish intellectual risk-taking, search, etc.?

While this is an empirical question beyond the scope of this post, it worth asking because it partially informs the micro-dynamics plausibly responsible for generating the spread of new intellectual paradigms.