Python For Research

Where Python is genuinely the right tool

The honest framing is that Python is a general-purpose language that happens to be excellent at scientific work, rather than a statistics package. That shapes what it is good for.

• Text at scale. Thousands of documents, transcripts, social media posts or open-ended responses. Cleaning, tokenising, topic modelling, sentiment classification and the use of language models all sit naturally here. Qualitative software will not process that volume, and statistics packages are not built for text.
• Collecting data from the web. Scraping pages, working with APIs, downloading structured data at intervals. Increasingly common as a doctoral data source, and Python is the standard tool.
• Automating repetitive work. Renaming and reorganising thousands of files, converting formats, applying the same processing to a hundred datasets. Hours of manual work become a script that runs once.
• Machine learning and prediction. Where the question is genuinely predictive rather than explanatory, the mature tooling is here.
• Images, audio and sensor data. Analysis of non-tabular data of almost any kind.
• Simulation. Including simulation-based power analysis for designs too complex for standard calculators.

What connects these is that none of them is a t-test. Where your analysis is a conventional statistical model reported in a conventional format, Python is capable but is rarely the path of least resistance.

Where it is the harder road

Python can run regressions, ANOVAs and mixed models through its statistical libraries. The question is whether it should, in a doctorate.

Three practical frictions:

Output is not in your discipline’s format. Statistics packages produce results laid out roughly as journals expect. Python gives you the numbers and leaves the presentation to you, which is more work at exactly the point when time is short.

Specialised methods may be missing or less mature. Structural equation modelling, advanced psychometrics and some multilevel specifications are better supported in R or in dedicated software. Discovering a required method is unavailable, mid-analysis, is an expensive way to learn this.

Fewer people around you will be able to help. In most social science, health and business faculties, more colleagues use SPSS, Stata or R than Python for statistical work.

A common and sensible arrangement is to use both: Python for collection, cleaning and text processing, then R or a statistics package for the modelling and reporting. Describing that division in your methods chapter is straightforward and reads as competence rather than complication.

Scraping: the questions you will be asked

Collecting data from the web is where doctoral candidates most often start writing Python, and where they most often proceed without considering questions an ethics committee will raise. Worth thinking through before you collect, not after.

• Are you permitted to collect it? Site terms of service, and the file most sites publish indicating what automated access they permit, are the starting point. An API, where one exists, is almost always the better route: it is sanctioned, more stable, and easier to justify.
• Is it personal data? Publicly visible does not mean unrestricted. Posts by identifiable individuals are usually personal data under data protection law regardless of being public, which brings obligations about storage, retention and what you may publish.
• Would participants expect to be studied? A recognised distinction in research ethics: a public discussion forum and a support group with an open door are technically both accessible and are not the same setting. Committees ask about this.
• How will you report it? A verbatim quotation from a public post is often searchable back to its author, which can identify someone who never consented to being studied. Paraphrasing, aggregating or seeking consent are the usual responses.
• Is your collection considerate? Requests at a reasonable rate, identifying your scraper honestly, and not degrading the service for others.

None of this rules scraping out — it is a legitimate and valuable method. It does mean the ethics application needs to address collection, storage and reporting specifically, and that it is far easier to get approval before you have collected anything than to explain afterwards.

Keeping it reproducible

Python shares R’s main advantage — the analysis is written down — and adds one specific risk that catches people out.

Record your environment. Python projects depend on many libraries, each with versions that change. An analysis that ran last year may not run today unless the versions are recorded. Standard tools exist for capturing exactly what your project used; one command produces the file, and it belongs with your thesis materials.

Notebooks are excellent for exploring and awkward for final analysis. Jupyter notebooks interleave code, output and notes, which suits developing an analysis. The hazard is that cells can be run in any order, so a notebook can show results that cannot be reproduced by running it from top to bottom. Before you rely on any result, restart and run the whole notebook in order. If it fails, what you had was not a working analysis.

Set a random seed anywhere randomness is involved — train and test splits, bootstrapping, model initialisation. Without it, numbers shift between runs, and a figure in your thesis stops matching your output.

Keep raw data untouched. Read it, process it in code, write results elsewhere. The same discipline as any other tool.

Reporting Python work

Name Python with its version and the libraries used for substantive work with theirs. Many scientific libraries have an associated paper the authors ask you to cite; using it is expected in fields where these tools are common.

Where you used machine learning, the reporting bar is specific and examiners in quantitative fields know it: how the data was split, what was tuned and on what, what was held out, and the performance measures with an honest account of what they mean. A model evaluated on the data it was tuned on is not evaluated.

Include your code as supplementary material. For computational work this is increasingly expected rather than generous, and it is the clearest possible evidence of what you did.

And describe the division of labour if you used more than one tool. “Data were collected and pre-processed in Python; statistical modelling was conducted in R” is a perfectly ordinary sentence and tells a reader exactly what happened.