Installing Social Feed Manager Locally

The easiest way to get started with Social Feed Manager is to install Docker on a local machine, such as a laptop or (preferably) desktop computer with a persistent internet connection.

Running SFM locally for anything other than testing purposes is NOT recommended. It will not be sufficient for a long-term documentation project and would typically allow only one local user to access the application’s web interface under localhost. But it could be useful for experimentation when access to a dedicated web server is not possible or desirable.

After testing the software on several Apple computers, I developed instructions for installing locally. Those who wish to install the application on locally should note that the virtualization software Docker must be installed and running on the host operating system and that no support for installing locally is provided, either by GW Libraries or by me.

One the application is running, it will be available at http://localhost/, under the port specified in the configuration (.env) file. So long as Docker and the application remain running and an Internet connection is live, the application will run harvests as scheduled in the user interface and will be available at localhost. The application can be stopped gracefully from the terminal at any time, with the command docker-compose stop.

To install SFM locally, a user would take the following steps:

  • Download and install Docker. There are instructions for Mac and for Linux Ubuntu on the project site, and Docker can be also installed for Linux and other operating systems. More information about Docker is in the sidebar.
  • Launch the docker software. Once it is running, you will see a little whale icon in your task or applications bar.
  • If it is not already installed on your computer, install git from Be sure to download the version appropriate to your operating system.
  • Open a terminal session (in linux/mac) or the command prompt in Windows.
  • Follow the local installation instructions at
    1. Use the terminal to clone SFM software from github and to place it in a dedicated folder on your computer. (Simply cut and paste the provided text into a terminal, then hit enter).
    2. Set configuration variables by editing the file .env, using a text editor of your choice. These variables are described fully in the documentation. Key variables that must be set for the application to be usable include the following:
      1. SFM_HOSTNAME and SFM_PORT. By setting these to ‘localhost’ and ‘8080’ respectively, the user interface will be available at http://localhost/ on your computer.
      2. SFM_SMTP_HOST, SFM_EMAIL_USER, and SFM_EMAIL_PASSWORD. Once the email variables are set correctly, SFM will email your reminders and password resets, as well as other notifications. If you are running SFM locally, you may use the credentials for a gmail or other account, but it is recommended that you create an account just for this applications, not use your personal settings since anyone with access to the server will be able to read your password in clear text.
      3. DATA_VOLUME and PROCESSING_VOLUME. As noted in the documentation, SFM’s default settings will save data inside the docker containers, where it is not accessible to you via the usual file system. If you would like direct access to the data, it can be saved on the usual file system. Follow the instructions in the documentation to set the storage location to a local volume or folder. For example:

        will set the storage location to the folder “sfm-data” on the root of your local file system, whereas:


        will set the sfm-data folder inside the docker containers, where it is accessible only through the docker commands for managing data in containers

    3. Launch the application using the command dockercompose up d
    4. Optionally, set multiple harvesting streams with the command:
      docker-compose scale twitterrestharvester=x

      where x is the number of simultaneous twitter harvesting streams that you wish to use. Please note that if SFM exceeds twitter’s data limits for a particular credential, it will likely be throttled or cut off.

Preserving Email Report Summary

Earlier today, I provided a summary of Preserving Email, a Technology Watch Report I wrote back in 2011.  I’ll leave it to others to judge how well that report holds up, but I had the following takeaways when re-reading it:

  • Email preservation has made great strides since then.  The range of tools now available to complete email archiving work is impressive and growing.
  • The same fundamental issues still impede our ability to acquire and preserve email: namely institutional and social mistrust of the archival function.
  • Given the, ahem, high level of interest that members of the public have in email controversies, and the relatively easy ways in which even public figures survive the disclosure of information from their public email account, we should be much more aggressive in acquiring and preserving email.  We really have nothing to lose.

While it may be of no use to anyone other than me, you can find a summary of my report here:

And the original report is found at:

Introducing Archival Connections

Welcome! This shares information from a five-year research project that I am coordinating at the University of Illinois at Urbana-Champaign.  The project aims to make it easier for people to find and use the materials managed by archival repositories like the University of Illinois Archives, where I work.  You can read more about the project on the about page.

This site is a successor to my Practical E-Records blog, but the focus here is less on preservation and more on access to born-digital, ‘cloud-based’ and other archival records.  Archival Connections is intended to be a collaborative effort, both inside Illinois and in the broader digital archives/Library community.