Install on your own computer

Choose your computer

Every instruction below changes to match this choice.

The one-step install

This script does everything on the rest of this page: finds or installs Miniforge, creates the surveys environment, installs the library, asks for your World Bank key and stores it in the right place, then reads real survey data to prove it worked. It is safe to run more than once — anything already done is skipped — and it does not touch your shell startup files.

curl -fsSL https://hhsurveys-workshop.ligonresearch.org/install_surveys.sh -o install_surveys.sh
bash install_surveys.sh

cd $HOME\Downloads
Invoke-WebRequest https://hhsurveys-workshop.ligonresearch.org/install_surveys.ps1 -OutFile install_surveys.ps1
powershell -ExecutionPolicy Bypass -File .\install_surveys.ps1

Read it first if you like — it is plain text and commented throughout: install_surveys.sh · install_surveys.ps1.

Or, the four steps by hand

1. Install Miniforge

   Miniforge gives you Python and a way to install scientific packages. If you already have Anaconda or Miniconda, you can skip this step and use that instead.

   Download and run the installer, accepting every default:

   Miniforge3-Windows-x86_64.exe

   When it finishes you will have a new program in the Start menu called Miniforge Prompt. Every command on this page gets typed there — not in Command Prompt or PowerShell.

   Open Terminal (press Cmd + Space, type terminal, press Return), then paste these three lines one at a time:

   curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-$(uname -m).sh"
   bash Miniforge3-MacOSX-$(uname -m).sh -b
   ~/miniforge3/bin/conda init zsh

   Then close Terminal and open it again. You should see (base) at the start of the line.

   Open a terminal and paste these three lines one at a time:

   curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-$(uname -m).sh"
   bash Miniforge3-Linux-$(uname -m).sh -b
   ~/miniforge3/bin/conda init bash

   Then close the terminal and open it again. You should see (base) at the start of the line.

2. Create the workshop environment

   This creates a separate workspace called surveys, so nothing here can disturb other software on your computer. Open the Miniforge Prompt from the Start menu and paste these two lines, one at a time:

   conda create -y -n surveys python=3.12 git
   conda activate surveys

   The first line downloads a few hundred megabytes and takes a while. When the second finishes, the line should begin with (surveys).

   This creates a separate workspace called surveys, so nothing here can disturb other software on your computer. Paste these two lines, one at a time:

   conda create -y -n surveys python=3.12 git
   conda activate surveys

   The first line downloads a few hundred megabytes and takes a while. When the second finishes, the line should begin with (surveys).

3. Install the survey library

   With (surveys) showing at the start of your line, paste:

   pip install LSMS_Library datamat jupyterlab

   This is the big one — about 1.1 GB and roughly 180 packages. Expect anywhere from a few minutes to the better part of an hour, depending on your connection. It is finished when you get your prompt back and see no red ERROR text.

4. Check that it worked

   Start Jupyter by pasting:

   jupyter lab

   A page opens in your browser. Click Notebook → Python 3, paste the five lines below into the box, and press Shift + Return.

   import lsms_library as ll
   
   ghana = ll.Country('GhanaLSS')
   print(ghana.waves)
   
   roster = ghana.household_roster()
   print(roster.shape)

   The first run downloads real survey data — about 34 MB. On a fast connection that takes a minute or two; on a slow one it can take considerably longer. Later runs take a couple of seconds, because the data is then stored on your machine.

   If it does not finish

   The years printing tells you only that the library loaded. What matters is what the cell does next, and there are two quite different possibilities:

   • The cell is still busy. Jupyter shows [*] beside it while it works. It is downloading the survey data — leave it running, even for ten minutes or more. This happens only on the first run.
   • You get a red Traceback. Read its last line, which names the real problem:
     • NoCredentialsError: Unable to locate credentials — no World Bank API key. See below; it is a five-minute fix, and free.
     • Bad git executable — git is missing; the surveys environment is probably not active.
     • Anything else — run the fuller check at the bottom of this page and send the two REPORT lines.

   On the workshop server these lines returned almost instantly, because the data was already staged there. On your own machine the first run downloads it, so give it time.

   What success looks like

   ['1987-88', '1988-89', '1991-92', '1998-99', '2005-06', '2012-13', '2016-17']
   (246594, 8)

   That is the Ghana Living Standards Survey: seven rounds spanning thirty years, and 246,594 people in the household rosters. If you see those two lines, you are done — please reply and say so.

   Harmless messages you can ignore

   Lines mentioning DVC unavailable, Falling back to manual aggregation, Issue with categorical mapping, or matplotlib building the font cache are all normal. They are not errors. Only the red Traceback blocks matter.

If something fails

These are the three failures we expect. In every case, the answer is to fix it if it is quick, and otherwise to stop and tell us — you will not be behind.

import yaml
from lsms_library import config

key = "PASTE-YOUR-KEY-HERE"

dst = config._config_file()
dst.parent.mkdir(parents=True, exist_ok=True)
data = yaml.safe_load(dst.read_text()) if dst.exists() else {}
data = {**(data or {}), "microdata_api_key": key}
dst.write_text(yaml.safe_dump(data, sort_keys=True))
print("saved to", dst)

Tell us how it went

Whether it worked or not, please send one line saying which computer you used and where you stopped. If you want to give us a more precise report, run the fuller check below instead of the five lines in step 4 — it tests each requirement separately and prints two lines to copy into your reply.

"""Workshop setup check: Analysis of Household Surveys, U. of Ghana.

Run this after installing.  It checks each requirement in turn and
prints a short report.  Nothing here changes your computer; it only
looks at what is already installed and downloads one small table.

Usage:
    python check_setup.py

or paste the whole file into a Jupyter cell and run it.
"""

import platform
import shutil
import sys

RESULTS = []


def record(name, ok, detail=""):
    RESULTS.append((name, ok, detail))
    mark = "PASS" if ok else "FAIL"
    line = f"[{mark}] {name}"
    if detail:
        line += f" -- {detail}"
    print(line, flush=True)
    return ok


def main():
    print("=" * 62)
    print("Analysis of Household Surveys -- setup check")
    print("=" * 62)
    print(f"System: {platform.system()} {platform.machine()}")
    print(f"Python: {sys.version.split()[0]}")
    print("-" * 62)

    # 1. Python version -------------------------------------------------
    ver = sys.version_info
    record(
        "Python 3.11 or newer",
        ver >= (3, 11),
        f"found {ver.major}.{ver.minor}.{ver.micro}"
        + ("" if ver >= (3, 11) else "; this is too old"),
    )

    # 2. git on PATH -----------------------------------------------------
    git = shutil.which("git")
    record(
        "git is installed",
        git is not None,
        git or "not found -- the library cannot even start without it",
    )

    # 3. gpg on PATH -----------------------------------------------------
    # Informational only.  LSMS_Library decrypts its own S3 read credentials
    # in pure Python (Fernet, via `cryptography', a hard dependency) as of
    # GH #741; the bundled s3_reader_creds.enc ships in the wheel.  The gpg
    # binary -- which macOS and Windows do not ship -- is now only a fallback
    # for wheels built before that file existed.  Reporting its absence as a
    # FAIL sent people off to install Gpg4win for no reason.
    gpg = shutil.which("gpg") or shutil.which("gpg2")
    record(
        "gpg is installed (optional)",
        True,
        gpg or "not found -- fine; decryption is pure Python since LSMS_Library 0.10",
    )

    # 3b. the World Bank Microdata API key --------------------------------
    # This is the real access gate: a valid key proves acceptance of the
    # World Bank's terms of use, and is what unlocks both direct downloads
    # and the shared S3 read cache.
    try:
        from lsms_library import config as _cfg

        _key = _cfg.microdata_api_key()
        record(
            "World Bank Microdata API key",
            bool(_key),
            (f"configured ({_key[:6]}...)" if _key
             else "not set -- data access will fail; see the install page"),
        )
    except Exception as exc:  # noqa: BLE001 -- pre-import, report anything
        record("World Bank Microdata API key", False, f"could not check: {exc}")

    # 4. the library imports ---------------------------------------------
    lib_ok = False
    try:
        import lsms_library as ll

        lib_ok = record(
            "lsms_library imports",
            True,
            f"version {getattr(ll, '__version__', 'unknown')}",
        )
    except Exception as exc:  # noqa: BLE001 -- report anything to the student
        record("lsms_library imports", False, f"{type(exc).__name__}: {exc}")

    # 4b. pandas matches what THIS lsms_library asks for -------------------
    # Do not hard-code a bound. 0.9.x needs pandas < 3; 0.10.1 and later need
    # pandas > 3. Read the requirement from the installed package's own
    # metadata so this check stays right across that change instead of
    # failing a correct installation.
    try:
        import pandas as _pd
        from importlib.metadata import requires as _requires
        from packaging.requirements import Requirement as _Req
        from packaging.version import Version as _V

        _spec = None
        for _r in _requires("LSMS_Library") or []:
            _req = _Req(_r)
            if _req.name.lower() == "pandas" and not _req.marker:
                _spec = _req.specifier
                break

        if _spec is None:
            record("pandas version suits the library", True,
                   f"pandas {_pd.__version__}; library states no bound")
        else:
            _ok = _V(_pd.__version__) in _spec
            record(
                "pandas version suits the library",
                _ok,
                f"pandas {_pd.__version__}, library wants pandas{_spec}"
                + ("" if _ok else '; run: pip install --upgrade LSMS_Library'),
            )
    except Exception as exc:  # noqa: BLE001
        record("pandas version suits the library", False,
               f"{type(exc).__name__}: {exc}")

    # 5. Jupyter ----------------------------------------------------------
    jupyter_exe = shutil.which("jupyter")
    jupyter_mod = False
    for mod in ("jupyterlab", "notebook", "jupyter_server"):
        try:
            __import__(mod)
            jupyter_mod = True
            break
        except Exception:  # noqa: BLE001
            continue
    record(
        "Jupyter is installed",
        bool(jupyter_exe or jupyter_mod),
        (jupyter_exe or "found as a Python package")
        if (jupyter_exe or jupyter_mod)
        else "not found -- run: pip install jupyterlab",
    )

    # 6. real data downloads ----------------------------------------------
    if lib_ok:
        print("-" * 62)
        print("Downloading a table from the Ghana Living Standards Survey.")
        print("This is the slow step -- one to several minutes on a first")
        print("run, depending on your connection.  Later runs are fast,")
        print("because the data is cached on your machine.")
        print("-" * 62, flush=True)
        try:
            import lsms_library as ll

            ghana = ll.Country("GhanaLSS")
            waves = ghana.waves
            record("Ghana survey waves found", bool(waves), f"{len(waves)} waves")
            roster = ghana.household_roster()
            record(
                "Ghana data downloads",
                len(roster) > 0,
                f"{len(roster):,} rows of household roster data",
            )
        except Exception as exc:  # noqa: BLE001
            record("Ghana data downloads", False, f"{type(exc).__name__}: {exc}")

    # Report ---------------------------------------------------------------
    print("=" * 62)
    failed = [name for name, ok, _ in RESULTS if not ok]
    if not failed:
        print("ALL CHECKS PASSED.  You are ready for the workshop.")
    else:
        print("SOME CHECKS FAILED:")
        for name in failed:
            print(f"  - {name}")
        print()
        print("This is normal and it is not a problem.  The workshop runs")
        print("on a shared server, so you can take part either way.")
    print()
    print("Please copy the two lines below into your reply:")
    print("-" * 62)
    print(f"REPORT {platform.system()}/{platform.machine()} py{ver.major}.{ver.minor}")
    print("REPORT " + ("all-passed" if not failed else "failed: " + ", ".join(failed)))
    print("-" * 62)


if __name__ == "__main__":
    main()