How to put Python and the survey library on your own laptop. This is optional preparation, not a requirement — but if it works, you will leave the workshop with a setup you can keep using.
You do not need to finish this to attend. The five sessions run on a shared server that you reach through a web browser, so everything we do in class will work even if nothing below succeeds. Try it anyway, and tell us how far you got — that is genuinely useful to us, and a failure report costs you nothing.
The download is roughly 1.3 GB, and the survey data adds a few hundred megabytes more. Please do not start it in the classroom or on mobile data — twenty laptops downloading at once will simply stall, and you will lose the session. Set aside an evening somewhere with a good connection.
Miniforge gives you Python and a way to install scientific packages. If you already have Anaconda or Miniconda, you can skip this step and use that instead.
Download and run the installer, accepting every default:
Open Terminal (press Cmd + Space,
type terminal, press Return), then paste these three
lines one at a time:
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-$(uname -m).sh" bash Miniforge3-MacOSX-$(uname -m).sh -b ~/miniforge3/bin/conda init zsh
Open a terminal and paste these three lines one at a time:
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-$(uname -m).sh" bash Miniforge3-Linux-$(uname -m).sh -b ~/miniforge3/bin/conda init bash
The library uses a tool called GnuPG to unlock its data downloads. On Windows it does not come with Miniforge, so it needs its own installer. Download Gpg4win and run it, accepting the defaults:
Now open the Miniforge Prompt from the Start menu and paste these two lines, one at a time:
conda create -y -n surveys python=3.12 git conda activate surveys
This creates a separate workspace called
surveys, so nothing here can disturb
other software on your computer. Paste these two lines, one at a
time:
conda create -y -n surveys python=3.12 git gnupg conda activate surveys
With (surveys) showing at the start of
your line, paste:
pip install LSMS_Library jupyterlab
Start Jupyter by pasting:
jupyter lab
A page opens in your browser. Click
Notebook → Python 3, paste the
five lines below into the box, and press
Shift + Return.
import lsms_library as ll
ghana = ll.Country('GhanaLSS')
print(ghana.waves)
roster = ghana.household_roster()
print(roster.shape)The years printing tells you only that the library loaded. What matters is what the cell does next, and there are two quite different possibilities:
[*] beside it while it works. It is
downloading the survey data — leave it running, even for
ten minutes or more. This happens only on the first run.
Traceback.
Read its last line, which names the real problem:
NoCredentialsError: Unable to locate
credentials — GnuPG is missing. This is the most
common one, especially on Windows. See below; it is a
five-minute fix.
Bad git executable — git is
missing; the surveys environment
is probably not active.
REPORT lines.
On the workshop server these lines return almost instantly, because the data and the tools are already in place there. Working on the server is unaffected by any of this.
['1987-88', '1988-89', '1991-92', '1998-99', '2005-06', '2012-13', '2016-17']
(246594, 8)That is the Ghana Living Standards Survey: seven rounds spanning thirty years, and 246,594 people in the household rosters. If you see those two lines, you are done — please reply and say so.
Lines mentioning DVC unavailable,
Falling back to manual aggregation,
Issue with categorical mapping, or
matplotlib building the font cache
are all normal. They are not errors. Only the red
Traceback blocks matter.
These are the three failures we expect. In every case, the answer is to fix it if it is quick, and otherwise to stop and tell us — you will not be behind.
The library needs Python 3.11 or newer. This usually means step 2
was skipped, or an older Anaconda is in charge. Check with
python --version; if it is below 3.11,
run step 2 again and make sure the line begins with
(surveys).
The library cannot even start without git.
It is installed in step 2, so this points to the same cause: the
surveys environment is not active.
Run conda activate surveys and try again.
The server has both the survey data and the tools already in place,
so it is not a fair comparison. On your own machine the last line is
the one that does real work, and it can fail for two quite different
reasons: it may simply still be downloading, or it may have stopped
with NoCredentialsError, which means
GnuPG is missing rather than anything to do with your account.
Check whether the cell still shows [*],
and read the last line of any red traceback.
This message is misleading: it almost always means
gpg is not installed, not that anything
is wrong with your account. On Windows, install
Gpg4win from step 2 and then close and reopen the Miniforge
Prompt.Run
conda install -y -c conda-forge gnupg
inside the surveys environment.
Installing GnuPG is the fix for this particular message. A World Bank Microdata key — which you should get in any case, see below — will also let the library reach the data, though by a slower route: it downloads from the World Bank directly rather than from the fast shared cache. With both GnuPG and a key you get the fast route and the full library.
Simplest of all, if you are stuck: log in to the
workshop server and open the folder
for-your-laptop in your home directory.
It contains one small file and a README explaining what to do with
it — copying that file onto your own machine restores the fast
route without GnuPG.
It is free, takes a few minutes, and we would encourage everyone to do it: microdata.worldbank.org. Register, then accept the terms of use for the LSMS collections.
Two reasons it is worth your while. It opens up the whole of the library rather than the slice staged for this workshop — roughly forty countries of survey data, which is what makes the tool useful for your own research afterwards. And it puts your access to the data on its proper footing: the surveys belong to national statistical offices and the World Bank, whose terms ask each user to accept them individually. Holding your own key is how you do that.
microdata.worldbank.org/auth/register. Give your name, email, institution and a password.
You cannot log in until you click that link. Check your spam folder — this is where most people get stuck.
Log in, then click the person icon with your name at the top right and choose Profile. Direct link: microdata.worldbank.org/index.php/auth/profile.
Scroll past your details to a section headed API keys. It will say “No API keys found.” Click the blue Generate API key button beside that heading.
The key — 64 characters — appears in a box with the warning that it will not be shown again. That is literal. Afterwards you see only its first few characters. If you lose it, delete it and generate another.
To save the key on your own machine, paste this into a Jupyter cell, putting your key between the quotes:
from pathlib import Path
import json
key = "PASTE-YOUR-KEY-HERE"
dst = Path.home() / ".config" / "lsms_library" / "config.yml"
dst.parent.mkdir(parents=True, exist_ok=True)
dst.write_text("microdata_api_key: " + json.dumps(key) + "\n")
print("saved to", dst)Treat the key like a password. It carries your identity and your permissions — anyone holding it acts as you. Never put it in a shared notebook, an email, or a git repository.
It expires after a year, and you may hold at most five at a time. Both are visible on the same profile page.
A bad or expired key reports
LOGIN_REQUIRED, not “bad
key”. If downloads start failing that way, generate a fresh
one rather than assuming you are signed out.
Whether it worked or not, please send one line saying which computer you used and where you stopped. If you want to give us a more precise report, run the fuller check below instead of the five lines in step 4 — it tests each requirement separately and prints two lines to copy into your reply.
"""Workshop setup check: Analysis of Household Surveys, U. of Ghana.
Run this after installing. It checks each requirement in turn and
prints a short report. Nothing here changes your computer; it only
looks at what is already installed and downloads one small table.
Usage:
python check_setup.py
or paste the whole file into a Jupyter cell and run it.
"""
import platform
import shutil
import sys
RESULTS = []
def record(name, ok, detail=""):
RESULTS.append((name, ok, detail))
mark = "PASS" if ok else "FAIL"
line = f"[{mark}] {name}"
if detail:
line += f" -- {detail}"
print(line, flush=True)
return ok
def main():
print("=" * 62)
print("Analysis of Household Surveys -- setup check")
print("=" * 62)
print(f"System: {platform.system()} {platform.machine()}")
print(f"Python: {sys.version.split()[0]}")
print("-" * 62)
# 1. Python version -------------------------------------------------
ver = sys.version_info
record(
"Python 3.11 or newer",
ver >= (3, 11),
f"found {ver.major}.{ver.minor}.{ver.micro}"
+ ("" if ver >= (3, 11) else "; this is too old"),
)
# 2. git on PATH -----------------------------------------------------
git = shutil.which("git")
record(
"git is installed",
git is not None,
git or "not found -- the library cannot even start without it",
)
# 3. gpg on PATH -----------------------------------------------------
gpg = shutil.which("gpg") or shutil.which("gpg2")
record(
"gpg is installed",
gpg is not None,
gpg or "not found -- data downloads will fail with a credentials error",
)
# 4. the library imports ---------------------------------------------
lib_ok = False
try:
import lsms_library as ll
lib_ok = record(
"lsms_library imports",
True,
f"version {getattr(ll, '__version__', 'unknown')}",
)
except Exception as exc: # noqa: BLE001 -- report anything to the student
record("lsms_library imports", False, f"{type(exc).__name__}: {exc}")
# 5. Jupyter ----------------------------------------------------------
jupyter_exe = shutil.which("jupyter")
jupyter_mod = False
for mod in ("jupyterlab", "notebook", "jupyter_server"):
try:
__import__(mod)
jupyter_mod = True
break
except Exception: # noqa: BLE001
continue
record(
"Jupyter is installed",
bool(jupyter_exe or jupyter_mod),
(jupyter_exe or "found as a Python package")
if (jupyter_exe or jupyter_mod)
else "not found -- run: pip install jupyterlab",
)
# 6. real data downloads ----------------------------------------------
if lib_ok:
print("-" * 62)
print("Downloading a table from the Ghana Living Standards Survey.")
print("This is the slow step -- one to several minutes on a first")
print("run, depending on your connection. Later runs are fast,")
print("because the data is cached on your machine.")
print("-" * 62, flush=True)
try:
import lsms_library as ll
ghana = ll.Country("GhanaLSS")
waves = ghana.waves
record("Ghana survey waves found", bool(waves), f"{len(waves)} waves")
roster = ghana.household_roster()
record(
"Ghana data downloads",
len(roster) > 0,
f"{len(roster):,} rows of household roster data",
)
except Exception as exc: # noqa: BLE001
record("Ghana data downloads", False, f"{type(exc).__name__}: {exc}")
# Report ---------------------------------------------------------------
print("=" * 62)
failed = [name for name, ok, _ in RESULTS if not ok]
if not failed:
print("ALL CHECKS PASSED. You are ready for the workshop.")
else:
print("SOME CHECKS FAILED:")
for name in failed:
print(f" - {name}")
print()
print("This is normal and it is not a problem. The workshop runs")
print("on a shared server, so you can take part either way.")
print()
print("Please copy the two lines below into your reply:")
print("-" * 62)
print(f"REPORT {platform.system()}/{platform.machine()} py{ver.major}.{ver.minor}")
print("REPORT " + ("all-passed" if not failed else "failed: " + ", ".join(failed)))
print("-" * 62)
if __name__ == "__main__":
main()