{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# An introduction to solving biological problems with Python\n", "\n", "## Session 2.4: Delimited files\n", "\n", "- [Data formats](#Data-formats)\n", "- [Exercises 2.4.1](#Exercises-2.4.1)\n", "- [Exercises 2.4.2](#Exercises-2.4.2)\n", "- [Exercises 2.4.3](#Exercises-2.4.3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data formats" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bioinformaticians love creating endless new file formats for their data, but there is one very common standard format\n", "that it is good to get used to parsing." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Delimited file example:\n", "```\n", "X 169008682 1 111267453 1.0976\n", "2 8265484 5 69763543 4.9825\n", "MT 10924 MT 81934 7.2357\n", "3 127 8 10908776 1.2509\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reading delimited files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can use the various string manipulation techniques covered earlier to process delimited files in a fairly straightforward way. Here we loop through a file with columns delimited by spaces, reading the data for each row into a list, and storing each of these lists into a main results list." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To view the an example of a delimited file, open a terminal window, go to the course directory, and print the content of the file using `cat` command or open it using your favourite editor:\n", "\n", "```bash\n", "cat data/mydata.txt\n", "```\n", "\n", "```\n", "Index Organism Score\n", "1 Human 1.076\n", "2 Mouse 1.202\n", "3 Frog 2.2362\n", "4 Fly 0.9853\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "results = []\n", "\n", "with open(\"data/mydata.txt\", \"r\") as data:\n", " header = data.readline()\n", " for line in data:\n", " results.append(line.split())\n", " \n", " \n", "print(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we show a slightly more complicated example where we are reading the results into a more convenient data structure, a list of dictionaries with the dictionary keys corresponding to the column headers and the values to the values from each line. We also convert the columns to an appropriate type as we go." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "results = []\n", "\n", "with open(\"data/mydata.txt\", \"r\") as data:\n", " header = data.readline()\n", " for line in data:\n", " idx, org, score = line.split()\n", " row = {'Index': int(idx), 'Organism': org, 'Score': float(score)}\n", " results.append(row)\n", " \n", "print(results)\n", "print('Score of first row:', results[0]['Score'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Writing delimited files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Writing out a delimited file is also straightforward using the `join` method. Here, as an example we will recreate our original file from above, but this time we will delimit the columns with a comma." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "mydata = [{'Organism': 'Human', 'Index': 1, 'Score': 1.076}, \n", " {'Organism': 'Mouse', 'Index': 2, 'Score': 1.202}, \n", " {'Organism': 'Frog', 'Index': 3, 'Score': 2.2362}, \n", " {'Organism': 'Fly', 'Index': 4, 'Score': 0.9853}]\n", "\n", "with open('data/mydata.csv', 'w') as output:\n", " # write a header\n", " header = \",\".join(['Index', 'Organism', 'Score'])\n", " output.write(header + \"\\n\")\n", " for row in mydata:\n", " line = \",\".join([str(row['Index']), row['Organism'], str(row['Score'])])\n", " output.write(line + \"\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To view the output file, open a terminal window, go to the course directory, and print the content of the file using `cat` command or open it using your favourite editor:\n", "\n", "```bash\n", "cat data/mydata.csv\n", "```\n", "\n", "```\n", "Index,Organism,Score\n", "1,Human,1.076\n", "2,Mouse,1.202\n", "3,Frog,2.2362\n", "4,Fly,0.9853\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Last but not least\n", "\n", "### A big thank you!\n", "\n", "### Remember...\n", "- Our course webpage: http://pycam.github.io\n", "- The Python website: https://www.python.org/ \n", "- To fill the course survey ;-)\n", "- To come to our next course 'Working with Python: functions and modules' and register at https://training.csx.cam.ac.uk/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercises 2.4.1\n", "\n", "Write a script that reads a tab delimited file which has 4 columns: gene, chromosome, start and end coordinates; that computes each gene's length and stores it into a dictionary; and writes the results into a new tab separated file. You can find a data file in ` data/genes.txt` directory of the course materials." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercises 2.4.2 \n", "\n", "Read the lyrics of Imagine by John Lennon, 1971 from the file in `data/imagine.txt`. Split the text into words. Print the total number of words, and the number of distinct words. Calculate the frequency of each distinct word and store the result into a dictionary. Print each distinct word along with its frequency. Find the most frequent word longer than 3 characters in the song, print it with its frequency.\n", "\n", "