Biol425 2017 and Monte Carlo Club: Difference between pages

From QiuLab
(Difference between pages)
Jump to navigation Jump to search
imported>Lab
No edit summary
 
imported>Weigang
mNo edit summary
 
Line 1: Line 1:
<center>'''Computational Molecular Biology''' (BIOL 425/790.49, Spring 2015)</center>
__FORCETOC__
<center>'''Instructor:''' Dr Konstantinos Krampis, Associate Professor of Biology</center>
==(This week) Feb 24, 2017 "Idiots" (Due 3/3/2017)==
<center>'''Room:'''1000G HN (10th Floor, North Building, Computer Science Department,  [http://www.geography.hunter.cuny.edu/tbw/CS.Linux.Lab.FAQ/department_of_computer_science.faq.htm Linux Lab FAQ])</center>
* Source: Paul Nahin (2000), "Dueling Idiots". Problem #2: "When Idiots Duel"
<center>'''Class Hours:''' Wednesdays 10:10 am-12:40 pm </center>
* Game: Idiots A and B decide to duel with a gun in the following way: They will insert a single bullet into the gun's cylinder. The cylinder has a total of 6 slots. Idiot A will spin the cylinder and shoot at B. If the gun doesn't fire, then A will give the gun to B, who will spin the cylinder and then shoot at A. This back-and-forth duel will continue until one fool shoots (and kills) the other.  
<center>'''Office Hours & Address:''' Hours: 11am-3pm on Wednesdays Address: Belfer Research Building, 413 East 69th Street (between York and 1st Avenue), New York, NY 10021. [https://www.google.com/maps/place/413+E+69th+St,+New+York,+NY+10021/@40.7655886,-73.9561743,17z/data=!3m1!4b1!4m2!3m1!1s0x89c258c3d235f76f:0x4f3d0d5d8a78fe6?hl=en Direction via Google Map]; </center>
* Questions: (1) what is the probability that A will win (and B dies); (2) What is the average number of trigger pulls before someone dies?
----
==Course Schedule (All Wednesdays)==
 
===February 3. Course overview & Unix tools===
* Course Overview. Lecture slides: [[File:Sildes-02-03-2016.pdf|thumbnail]]
* Learning Goals: (1) Understand the "Omics" files; (2) Review/Learn Unix tools
* In-Class Tutorial: Unix file filters
# ''Without changing directory'', long-list genome, transcriptome, and proteome files using <code>ls</code>
# ''Without changing directory'', view genome, transcriptome, and proteome files using <code>less -S</code>
# Using <code>grep</code>, find out the size of ''Borrelia burgdorferi'' B31 genome in terms of the number of replicons ("GBB.1con" file)
# Using <code>sort</code>, sort the replicons by (a) contig id (3rd field, numerically); and (b) replicon type (4th field)
# Using <code>cut</code>, show only the (a) contig id (3rd field); (b) replicon type (4th field)
# Using <code>tr</code>, (a) replace "_" with "|"; (b) remove ">"
# Using <code>sed</code>, (a) replace "Borrelia" with abbreviation "B.";  (b) remove "plasmid" from all lines
# Using <code>paste -s</code>, extract all contig ids and concatenate them with ";"
# Using a combination of <code>cut and uniq</code>, count how many circular plasmids ("cp") and how many linear plasmids ("lp")
* In-Class Challenges
# Using the "GBB.seq" file, find out the number of genes on each plasmid
<div class="toccolours mw-collapsible">
<syntaxhighlight lang=bash">
grep ">" ../../bio425/data/GBB.seq | cut -c1-4| sort | uniq -c
</syntaxhighlight>
</div>
# Using the "ge.dat" file, find out (a) the number of genes; (b) the number of cell lines; (c) the expression values of three genes: ERBB2, ESR1, and PGR
<div class="toccolours mw-collapsible">
<syntaxhighlight lang=bash">
grep -v "Description" ../../bio425/data/ge.dat | wc -l; or:  grep -vc "Description" ../../bio425/data/ge.dat
grep "Description" ../../bio425/data/ge.dat | tr '\t' '\n'| grep -v "Desc" | wc -l
grep -Pw "ERBB2|PGR|ESR1" ../../bio425/data/ge.dat
</syntaxhighlight>
</div>
 
===Reading Material===
Lecture slides: [[File:Sildes-02-03-2016.pdf|thumbnail]] <br>
Unix 1a: [[File:Unix_1-smalla.pdf|Unix 1a ]]<br>
Unix 1b: [[File:Unix_1-smallb.pdf|Unix 1b ]]<br>
Unix 2: [[File:Unix-2-smallester.pdf|Unix 2 ]]<br>
Unix 3: [[File:Unix-3.pdf|Unix 3 ]]


==(Last week) Feb 17, 2017 "Birthday" (Due 2/24/2017)==
[[File:B-day.png|thumbnail]]
* Problem: What is the probability NONE of the N people in a room sharing a birthday?
# Randomly select N individuals and record their B-days
# Count the B-days NOT shared by ANY two individuals
# Repeat (for each N) 1000 times, and obtain probability by averaging the previous counts (i.e., divided by 1000)
# Vary N from 10 to 100, increment by 10
# Plot probability of no-shared B-Day (Y-axis) versus N (x-axis), with either a stripchart or boxplot, or both
{| class="wikitable sortable mw-collapsible"
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
|- style="background-color:lightsteelblue;"
! Assignment #1 - (<font color="red">Due Wednesday 2/10/2016, at 10am - Submit Responses on Blackboard</font>)
! Submitted Codes
|- style="background-color:powderblue;"
|- style="background-color:powderblue;"
| '''Unix Text Filters''' (10 pts) Show both commands and outputs for the following questions:<br>
|  
|Tip1: piping to the wc command (read the help for it !) allows you to do line counts <br>
* By weigang
Tip2: read the help for head command (what does the -n operator do?). If you specify head to get 1000 lines for example, you can then use tail command to get the last 10 of those 1000<br>
<syntaxhighlight lang="bash">
Tip3: Reading material should tell you what the tr command does, but also look at tr --help. Can you use cut instead of tr ?<br>
days <- 1:365;
# Without changing directory (i.e., remain in your home directory), locate and long-list the genbank file named "GBB.gb" in the course data directory
# Count the total number of lines, show the first and last 10 lines of the file. Using a combination of <code>head</code> and <code>tail</code> commands, show only the lines containing the translated protein sequence of the first gene
# Count the total number of replicans by extracting lines containing "LOCUS" (case sensitive); sort them by the total number of bases ("bp")
# Remove the string "(plasmid" from the above output
# Extract the second column (replicon names) from the above output. [Hint: these fields are delimited by an unequal number of spaces, not by tabs. Use <code>tr -s</code> to first squeeze to single space]
|}
----


=== February 10. Genomics (1): Gene-Finding===
find.overlap <- function(x) { return(length(which(table(x)>1))) }
* Learning goals: (1) Running UNIX programs; (2) Parse text with Perl anonymous hash
* Lecture slides: [[File: Slides-2-10-2016.pdf|thumbnail]]
* In-Class Tutorials
# Identify ORFs in a prokaryote genome
## Go to [http://www.ncbi.nlm.nih.gov/gorf/gorf.html NCBI ORF Finder page]
## Paste in the GenBank Accession: AE000791.1 and click "orfFind"
## Change minimum length for ORFs to "300" and click "Redraw". How many genes are predicted? What is the reading frame for each ORF? Coordinates? Coding direction?
## Click "Six Frames" to show positions of stop codons (magenta) and start codons (cyan)
# Gene finder using GLIMMER
## Locate the GLIMMER executables: <code>ls /data/biocs/b/bio425/bin/</code>
## Locate ''Borrelia'' genome files: <code>ls /data/biocs/b/bio425/data/GBB.1con-splitted/</code>
## Predict ORFs: <code>../../bio425/bin/long-orfs ../../bio425/data/GBB.1con-splitted/Borrelia_burgdorferi_4041_cp9_plasmid_C.fas cp9.coord</code> [Note the two arguments: one input file and the other output filename]
## Open output file with <code>cat cp9.coord</code>. Compare results with those from NCBI ORF Finder.
## Extract lines with coordinates" <code>grep "^[0-9]" cp9.coord > cp9.coord2</code>
## Extract sequences into a FASTA file: <code>../../bio425/bin/extract ../../bio425/data/GBB.1con-splitted/Borrelia_burgdorferi_4041_cp9_plasmid_C.fas cp9.coord2 > cp9.fas</code> [Note two input files and standard output, which is then redirected (i.e., saved) into a new file]
# bioseq
## Add <code>bp-utils</code> into $PATH by editing the .bash_profile file and run <code>source .bash_profile</code>
## Run <code>bioseq</code> with these options: <code>-l, -n, -c, -r, -t1, -t, -t6, and -s</code>
## In-Class Challenge: use <code>bioseq</code> to extract and translate the 1st (+1 frame) and 5th (-1 frame) genes.
<div class="toccolours mw-collapsible">
<syntaxhighlight lang=bash">
cp ../../bio425/data/GBB.1con-splitted/Borrelia_burgdorferi_4041_cp9_plasmid_C.fas cp9.plasmid.fas # copy plasmid sequence
bioseq -s'163,1272' cp9.plasmid.fas | bioseq -t1 # extract first predicted ORF (+1 frame)
bioseq -s'3853,4377' cp9.plasmid.fas | bioseq -r | bioseq -t1 # extract 5th ORF (since it is -1 frame, need to rev-com before translation
</syntaxhighlight>
</div>
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
! Assignment #2 (<font color="red">Due Wednesday 2/17/2016, at 10am - Submit Responses on Blackboard</font>)
|- style="background-color:powderblue;"
| '''UNIX Exercises and Reading Material''' (10 pts)<br>
[https://www.dropbox.com/sh/2s1ta3xdz1zzxhw/AAD70Mjhl15Gv5zJ3SwAogTea?dl=0 Lecture Videos]
* Some More Unix and Vi / Vim Editor: [[File: Feb-10-2016-reading-part-1.pdf|thumbnail]]
* Vi / Vim tutorial : http://www.openvim.com/tutorial.html  http://www.openvim.com/sandbox.html
* Unix environment variables and bashrc: [[File: Feb-10-2016-reading-part2.pdf|thumbnail]]
* Gene finding: [[File: Feb-10-2016-reading-part3.pdf|thumbnail]]
* Open Reading Frames: [[File: Feb-10-2016-reading-part4.pdf|thumbnail]]
* bioseq documentation and help [[Bioutils]]
# Using ssh, log into eniac.cs.hunter.cuny.edu & then ssh into another host (e.g. "cslab8") for the exercises - REQUIRED IF YOU ARE WORKING REMOTELY!
# Copy data file <code>../../bio425/data/mystery_seq1.fas</code> into your home directry
# Add <code>/data/biocs/b/bio425/bin</code> to your the $PATH variable in the ".bash_profile" file (using vi or emacs).
# Run <code>source ~/.bash_profile</code> to activate the above path
# Use the <code>long-orf</code> command to predict ORFs on the mystery_seq1. Save resulting coord file as "mystery_seq1.coord".
# Extract ONLY the lines containing ORF information into a new file "mystery_seq1.coord2" using <code>grep</code>
# Run <code>extract</code> to extract FASTA sequences of all predicted ORFs. Save as "mys.nuc".
# Translate "mys.nuc" with <code>bioseq -t1</code>. Save to "mys.pep" (don't be surprised if the 2nd sequence contains internal stop codons)  
# Use <code>bioseq</code> to extract the ORF sequences, save as "mys.nuc2"
# Translate "mys.nuc2" with <code>bioseq -t1</code>. Save to "mys.pep2". This file should NOT contain any internal stop codons.
Note: Show all commands and outputs.To log off, type "exit" twice (first log out of the "cslab" host, 2nd time log off the "eniac" host).
|}
----


===February 17. Genomics (2): BASH & BLAST===
output <- sapply(1:100, function(x) { # num of people in the room
* Lecture Slides: [[File:02-17-2015.pdf|thumbnail]]
  ct.no.overlap <- 0;
* Learning goal: (1) BASH scripting; (2) Homology searching with NCBI blast
  for (k in 1:100) {
* In-Class exercises
    bdays <- sample(days, x, replace = T);
# Build workflow with BASH scripts
    ct <- find.overlap(bdays);
<syntaxhighlight lang=bash">
    if (!ct) { ct.no.overlap <- ct.no.overlap + 1}
#!/usr/bin/bash
  }
bioseq -s"163,1272" cp9.plasmid.fas > gene-0001.nuc;
  return(ct.no.overlap);
bioseq -t1 gene-0001.nuc > gene-0001.pep;
})
exit;
</syntaxhighlight>
plot(1:100, output/100, xlab="Group size", ylab = "Prob (no shared b-day)", las=1, main="B-day (sim=100 times)", type="l")
Save the above as "extract-gene-v1.bash". Make it executable with <code>chomd +x extract-gene-v1.bash</code>. Run it with <code>./extract-gene-v1.bash</code>
abline(h=0.5, lwd=2, col=2, lty=2)
# Improvement 1. make it work for other genes (by using variables)
abline(v=seq(0, 100, 5), col="gray")
<syntaxhighlight lang=bash">
#!/usr/bin/bash
id=$1;
begin=$2;
end=$3;
bioseq -s"$begin,$end" cp9.plasmid.fas > gene-$id.nuc;
bioseq -t1 gene-$1.nuc > gene-$id.pep;
exit;
</syntaxhighlight>
Save the above as "extract-gene-v2.bash". Make it executable with <code>chomd +x extract-gene-v2.bash</code>. Run it with <code>./extract-gene-v2.bash 0001 163 1272</code> (Note three arguments). Question: How to modify the code to make it work for genes encoded on the reverse strand?
# Improvement 2. make it work for both positive and negative genes (by using an "if" statement)
<syntaxhighlight lang=bash">
#!/usr/bin/bash
id=$1;
begin=$2;
end=$3;
strand=$4;
bioseq -s"$begin,$end" cp9.plasmid.fas > gene-$id.nuc;
if [ $strand -gt 0 ]; then
    bioseq -t1 gene-$1.nuc > gene-$id.pep;
else
    bioseq -r gene-$1.nuc > gene-$id.rev;
    bioseq -t1 gene-$1.rev > gene-$id.pep;
fi;
exit;
</syntaxhighlight>
# Improvement 3. Make it read "cp9.coord" as an input file, by using a "while" loop:
<syntaxhighlight lang=bash">
#!/usr/bin/bash
cat $1 | grep "^[0-9]" | while read line; do
    id=$(echo $line | cut -f1 -d' '); # run commands and save result to a variable
    x1=$(echo $line | cut -f2 -d' ');
    x2=$(echo $line | cut -f3 -d' ');
    strand=$(echo $line | cut -f4 -d' ');
    if [ $strand -lt 0 ]; then
        bioseq -s"$x2,$x1" cp9.plasmid.fas > gene-$id.nuc;
        bioseq -r gene-$id.nuc > gene-$id.rev;
        bioseq -t1 gene-$id.rev;
    else
        bioseq -s"$x1,$x2" cp9.plasmid.fas > gene-$id.nuc;
        bioseq -t1 gene-$id.nuc;
    fi;
done;
exit;
</syntaxhighlight>
# Run Stand-alone BLAST:
* Add blast binaries to your $PATH in ".bash_profile": export PATH=$PATH:/data/biocs/b/bio425/ncbi-blast-2.2.30+/bin/
* BLAST tutorial 1. A single unknown sequence against a reference genome
<syntaxhighlight lang=bash" line enclose="div">
cp ../../bio425/data/GBB.pep ~/. # Copy the reference genome
cp ../../bio425/data/unknown.pep ~/. # Copy the query sequence
makeblastdb -in GBB.pep -dbtype prot -parse_seqids -out ref # make a database of reference genome
blastp -query unknown.pep -db ref # Run simple protein blast
blastp -query unknown.pep -db ref -evalue 1e-5 # filter by E values
blastp -query unknown.pep -db ref -evalue 1e-5 -outfmt 6 # concise output
blastp -query unknown.pep -db ref -evalue 1e-5 -outfmt 6 | cut -f2 > homologs-in-ref.txt # save a list of homologs
blastdbcmd -db ref -entry_batch homologs-in-ref.txt > homologs.pep # extract homolog sequences
</syntaxhighlight>
* In-Class challenges
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
! Assignment #3 (<font color="red">Due Wednesday 2/24/2016, at 10am - Submit Responses on Blackboard</font>)
|- style="background-color:powderblue;"
| '''Assignment''' (10 pts)<br>
IMPORTANT: Log on your eniac account and then ssh into a "cslab" host (e.g., ssh cslab5). Perform the following tasks without changing directory (i.e., stay in your home directory).
 
* [https://www.dropbox.com/sh/xttuu00538kaaxf/AAB2-k6UskwtGj0DhJMxu20Oa?dl=0 Lecture Videos]
* Reading Material: [[File:02-17-2016-1.pdf|thumbnail]]
* Reading Material: [[File:02-17-2016-2.pdf|thumbnail]]
* Reading Material: [[File:02-17-2016-3.pdf|thumbnail]]
* Reading Material: [[File:02-17-2016-4.pdf|thumbnail]]
* Reading Material: [[File:02-17-2016-BLAST-1.pdf|thumbnail]]
* Reading Material: [[File:02-17-2016-BLAST-2.pdf|thumbnail]]
 
 
# UNIX filter exercises (4 pts)
## For the codon file <code>/data/biocs/b/bio425/data/codon.table.T</code>, sort by single-letter code of amino acids
## use "grep" to show only codons that start with an "A"
## Show all codons for "Arg", without amino acid symbols
## Use an editor, compose a BASH script (name it "read-codon.bash") that uses a "while" loop to read each line of this file. First, use "grep -v" to exclude stop codons. Then, create three variables to capture the values of three elements on each line. For each line, print the output in the following format: "AAA codes for Lys (K)".
# Exercise for the use of wildcard & BASH "for" loop ( 3 pts)
## Long-list (i.e., ls -l) the following file in the "/data/biocs/b/bio425/data/GBB.1con-splitted/" directory: Borrelia_burgdorferi_3615_main.fas (this file contains the main chromosome sequence of the B31 strain of Lyme disease bacteria). Run <code>bioseq -l</code> to find out the length of this sequence.
## Without changing directory, long-list ALL files with the ".fas" suffix in the same directory (using wildcard). This command list all replicons (sequences in the .fas files, which are chromosome and 21 plasminds) of this genome.
## Write a BASH "for" loop on the command line (do not use an editor) to obtain length of each replicon using <code>bioseq -l</code>
# BLAST exercise (3 pts).
## Use "blastp" to identify matches of the sequence <code>../../bio425/data/unknown.pep</code> in the "ref" database (which was created in class), using an e-value cutoff of 1e-10
## Show all pairwise alignments
## Show output in a tabular format (using "-outfmt 6")
## Explain the following blast terms: "Identity", "Positives", and "Expect". [Hint, visit [http://blast.ncbi.nlm.nih.gov/ the NCBI BLAST home page] and [http://blast.ncbi.nlm.nih.gov/Blast.cgi?CMD=Web&PAGE_TYPE=BlastDocs&DOC_TYPE=FAQ#expect this FAQ]].
|}
 
===March 2. Genomics (3). Genome BLAST & BioPerl===
* Lecture Slides: [[File:02-24-2016.pdf|thumbnail]]
* Learning goals: Homology, BLAST, & Object-oriented programming with Perl
 
*BLAST tutorial 1. A single unknown sequence against a reference genome.
*Run Stand-alone BLAST:
* Add blast binaries to your $PATH in ".bash_profile": export PATH=$PATH:/data/biocs/b/bio425/ncbi-blast-2.2.30+/bin/
<syntaxhighlight lang=bash" line enclose="div">
cp ../../bio425/data/GBB.pep ~/. # Copy the reference genome
cp ../../bio425/data/unknown.pep ~/. # Copy the query sequence
makeblastdb -in GBB.pep -dbtype prot -parse_seqids -out ref # make a database of reference genome
blastp -query unknown.pep -db ref # Run simple protein blast
blastp -query unknown.pep -db ref -evalue 1e-5 # filter by E values
blastp -query unknown.pep -db ref -evalue 1e-5 -outfmt 6 # concise output
blastp -query unknown.pep -db ref -evalue 1e-5 -outfmt 6 | cut -f2 > homologs-in-ref.txt # save a list of homologs
blastdbcmd -db ref -entry_batch homologs-in-ref.txt > homologs.pep # extract homolog sequences
</syntaxhighlight>
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
* BLAST tutorial 2.
*Find homologs within the new genome itself
<syntaxhighlight lang=bash" line start=9 enclose="div">
cp ../../bio425/data/N40.pep ~/. # Copy the unknown genome
makeblastdb -in N40.pep -dbtype prot -parse_seqids -out N40 # make a database of the new genome
blastp -query unknown.pep -db N40 -evalue 1e-5 -outfmt 6 | cut -f2 > homologs-in-N40.txt # find homologs in the new genome
blastdbcmd -db N40 -entry_batch homologs-in-N40.txt >> homologs.pep # append to homolog sequences
</syntaxhighlight>
* BLAST tutorial 3.
*Multiple alignment & build a phylogeny
<syntaxhighlight lang=bash" line start= 13 enclose="div">
../../bio425/bin/muscle -in homologs.pep -out homologs.aln # align sequences
cat homologs.aln | tr ':' '_' > homologs2.aln
../../bio425/bin/FastTree homologs2.aln > homologs.tree # build a gene tree
../../bio425/figtree &  # view tree (works only in the lab; install your own copy if working remotely)
</syntaxhighlight>
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
!Assignment #4 (<font color="red">Due Wednesday 3/9/2016, at 10am - Submit Responses on Blackboard</font>)
|-style="background-color:powderblue;"
|'''BLAST exercise''' (5 pts)<br>
* Reading Material: [[File:02-17-2016-BLAST-1.pdf|thumbnail]]
* Reading Material: [[File:02-17-2016-BLAST-2.pdf|thumbnail]]
* [http://www.woolfit.net/perl/classindex.html Useful Perl tutorial for future classes and homework !]
* Reading Material: [[File:Perl-Strings.pdf|thumbnail]]
* Reading Material: [[File:Perl-Lists-Hashes.pdf|thumbnail]]
* Reading Material: [[File:Perl-Subroutines.pdf|thumbnail]]
* Reading Material: [[File:Perl-Basic-1.pdf|thumbnail]]
* Reading Material: [[File:Perl-Basic-2.pdf|thumbnail]]
 
* Perl : [https://www.youtube.com/watch?v=IoLVCEr207w a 2.5hr video tutorial !]
Run BLASTp to identify all homologs of BBA18 in the reference genome ("ref").  
# Obtain BBA18 peptide sequence using <code>blastdbcmd -db ref -entry 'BBA18'</code>.
# Run blastp against the "ref" database with the following e-value cutoffs: 1e-5, 1e-3, and 1e-1. Submit the results for 1e-3 with the "-outfmt 6" option.
# Extract all homologs using <code>blastdbcmd</code>
# Combine all sequences with <code>cat</code> and align all sequences using <code>muscle</code>. Show alignment.
# Build a tree of homologs using <code>FastTree</code>. Display tree with [http://www.evolgenius.info/evolview/ evolview].
''Perl exercise''' (5 pts)<br>
# Compose a Perl script shown below ("dump-coords.pl" in our files), which uses an array (@genes) to hold information for two genes in the <code>../../bio425/data/mys.coord2</code> file. Each element of the array would be a reference to a hash. The hash reference itself would hold gene information in the following format: e.g., <code>$gene={'id'=>'0001', 'start'=>16, 'end'=>324, 'frame'=>'+1', 'score'=>0.929}</code>. Print the array by using the <code>Dumper</code> function.
Do not hard-code the values of the hash within the code only the keys (id, start, stop, frame, score), the script must parse the data for the values from the file!
Tip: read elements of line into array, then construct anonymous hash (with $ not %) hashes using values from the elements of the array, and then push the anonymous hash to the @genes array. Remember you will need an intermediate array which will hold the information for the elements of each line, and will be "reloaded with each line.
Use the following skeleton code :
<syntaxhighlight lang=perl" enclose="div">
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper; # print complex data structure
# ----------------------------------------
# Author          : WGQ
# Date            : February 30, 2015
# Description    : Read coord file
# Input          : A coord file
# Output          : Coordinates and read frame for each gene
# ----------------------------------------
my @genes; # declare the array
while(<>) { # this means that as long as lines come from the pipe we keep going
  my $line = $_; # a line that come from the pipe (we go line by line)
  next unless $line =~ /^\d+/; # skip lines except those reporting genes
  <Your code: split the $line on white spaces and save into variables>
  <Your code: construct anonymous hash and push into @genes>
}
print Dumper(\@genes);
exit;
</syntaxhighlight>
|}
 
===March 9: Genomics (4). Object-oriented Perl (Continued) ===
 
* Lecture Slides: [[File:File-03-09-partA.pdf|thumbnail]]
* Lecture Slides: [[File:File-03-09-partB.pdf|thumbnail]]
 
*Reading material:
* [http://www.woolfit.net/perl/63bioseq.html Perl References 1]
* [http://search.cpan.org/dist/BioPerl/Bio/Seq.pm Perl References and Bio::Seq module]
* [http://www.woolfit.net/perl/classindex.html Perl Overview]
 
* Learning goal: (1) Object-Oriented Perl; (2) BioPerl
* In-Class Exercises
Construct and dump a Bio::Seq object
<syntaxhighlight lang="perl" line enclose="div">
#!/usr/bin/perl -w
use strict;
use lib '/data/biocs/b/bio425/bioperl-live';
use Bio::Seq;
use Data::Dumper;
my $seq_obj = Bio::Seq->new( -id => "ospC", -seq =>"tgtaataattcaggaaaaga" );
print Dumper($seq_obj);
exit;
</syntaxhighlight>
Apply Bio::Seq methods:
<syntaxhighlight lang="perl" line enclose="div">
my $seq_rev=$seq_obj->revcom()->seq(); # reverse-complement & get sequence string
my $eq_length=$seq_obj->length();
my $seq_id=$seq_obj->display_id();
my $seq_string=$seq_obj->seq(); # get sequence string
my $seq_translate=$seq_obj->translate()->seq(); # translate & get sequence string
my $subseq1 = $seq_obj->subseq(1,10); # subseq() returns a string
my $subseq2= $seq_obj->trunc(1,10)->seq(); # trunc() returns a truncated Bio::Seq object
</syntaxhighlight>
* Challenge 1: Write a BioPerl-based script called "bioperl-exercise.pl". Start by constructing a Bio::Seq object using the "mystery_seq1.fas" sequence. Apply the <code>trunc()</code> method to obtain a coding segment from base #308 to #751. Reverse-complement and then translate the segment. Output the translated protein sequence.
<syntaxhighlight lang="perl" line enclose="div">
#!/usr/bin/perl -w
use strict;
use lib '/data/biocs/b/bio425/bioperl-live';
use Bio::Seq;
my $seq_obj = Bio::Seq->new( -id => "mystery_seq", -seq =>"tgtaataattcaggaaaaga.............." );
print $seq_obj->trunc(308, 751)->revcom()->translate()->seq(), "\n";
exit;
</syntaxhighlight>
* Challenge 2. Re-write the above code using Bio::SeqIO to read the "mystery_seq1.fas" sequence and output the protein sequence.
<syntaxhighlight lang="perl" line enclose="div">
#!/usr/bin/perl -w
use strict;
use lib '/data/biocs/b/bio425/bioperl-live';
use Bio::SeqIO;
die "$0 <fasta_file>\n" unless @ARGV == 1;
my $file = shift @ARGV;
my $input = Bio::Seq->new( -file => $file, -format =>"fasta" );
my $seq_obj = $input->next_seq();
print $seq_obj->trunc(308, 751)->revcom()->translate()->seq(), "\n";
exit;
</syntaxhighlight>
</syntaxhighlight>
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
! Assignment #5 (<font color="red">Due Wednesday 3/16/2016, at 10am - Submit Responses on Blackboard</font>)
|- style="background-color:powderblue;"
| '''BioPerl exercises (10 pts)'''<br />
Perl exercise 1 (if you have submitted this before, please resubmit your earlier answer and note that is a resubmission) (5 pts)<br>
# Compose a Perl script shown below ("dump-coords.pl" in our files), which uses an array (@genes) to hold information for two genes in the <code>../../bio425/data/mys.coord2</code> file. Each element of the array would be a reference to a hash. The hash reference itself would hold gene information in the following format: e.g., <code>$gene={'id'=>'0001', 'start'=>16, 'end'=>324, 'frame'=>'+1', 'score'=>0.929}</code>. Print the array by using the <code>Dumper</code> function.
Do not hard-code the values of the hash within the code only the keys (id, start, stop, frame, score), the script must parse the data for the values from the file!
Tip: read elements of line into array, then construct anonymous hash (with $ not %) hashes using values from the elements of the array, and then push the anonymous hash to the @genes array. Remember you will need an intermediate array which will hold the information for the elements of each line, and will be "reloaded with each line.
Use the following skeleton code :
<syntaxhighlight lang=perl" enclose="div">
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper; # print complex data structure
# ----------------------------------------
# Author          : WGQ
# Date            : February 30, 2015
# Description    : Read coord file
# Input          : A coord file
# Output          : Coordinates and read frame for each gene
# ----------------------------------------
my @genes; # declare the array
while(<>) { # this means that as long as lines come from the pipe we keep going
  my $line = $_; # a line that come from the pipe (we go line by line)
  next unless $line =~ /^\d+/; # skip lines except those reporting genes
  <Your code: split the $line on white spaces and save into variables>
  <Your code: construct anonymous hash and push into @genes>
}
print Dumper(\@genes);
exit;
</syntaxhighlight>
Perl exercise 2 (5 pts)<br>
# Write an "extract.pl" using Perl & BioPerl that extract coding sequences from "coord" files. The script will use Bio::SeqIO to read the genome FASTA file ("mystery_seq1.fas", see Assignment #2 above). It will also read the "mystery_seq1.coord" (see Assignment #2 above) as the 2nd argument. Use Bio::Seq to obtain coding sequences and translate sequences. Your output of protein sequences should not contain stop codons except as the last codon. The following template helps you get started:
<syntaxhighlight lang="perl" line enclose="div">
#!/usr/bin/perl
use strict;
use warnings;
use lib '/data/biocs/b/bio425/bioperl-live';
use Bio::SeqIO;
# ----------------------------------------
# File            : extract.pl
# Author          : WGQ
# Date            : March 5, 2015
# Description    : Emulate glimmer EXTRACT program
# Input          : A FASTA file with 1 DNA seq and coord file from LONG-ORF
# Output          : A FASTA file with translated protein sequences
# ----------------------------------------
die "Usage: $0 <FASTA_file> <coord_file>\n" unless @ARGV > 0;
my ($fasta_file, $coord_file) = @ARGV;
my $fasta_input = Bio::SeqIO->new(<your code>); # create a file handle to read sequences from a file
my $output = Bio::SeqIO->new(-file=>">$fasta_file".".out", -format=>'fasta'); # create a file handle to output sequences into a file
my $seq_obj = $input->next_seq(); # get sequence object from FASTA file
# Read COORD file & extract sequences
open COORD, "<" . $coord_file;
while (<COORD>) {
      my $line = $_;
      chomp $line;
      next unless $line =~ /^\d+/; # skip lines except
      my ($seq_id, $cor1, $cor2, $strand, $score) = split /\s+/, $line; # split line on white spaces
      if (<your code: if strand is positive>) {
        <your code: use trunc function to get sub-sequence as an object>
      } else {
        <your code: use the trunc() function to get sub-sequence as an object>
        <your code: use the revcom() function to reverse-complement the seq object>
      }
      <your code: use translate() function to obtain protein sequence as an abject, "$pro_obj">
      $output->write_seq($pro_obj);
}
close COORD;
exit;
</syntaxhighlight>
|}
|}


===March 16: Course Material Practice and Review - Questions ===
==(Previous week) Feb 10, 2017 "Dating" (Valentine's Day Special; Due 2/17/2017)==
* Review slides: [[File:Review-Session.pdf|thumbnail]]
[[File:Dating.png|thumbnail]]
* [https://www.dropbox.com/sh/fdy60u8nlzgim8f/AACm6vX2hdhg-JGi7ZeZUyKCa?dl=0 BioPerl Lecture videos]
* Source: Paul Nahin (2008), "Digital Dice". Problem #20: "An Optimal Stopping Problem"
* [http://search.cpan.org/dist/BioPerl/Bio/SeqIO.pm Perl Doc 1]
* Problem: What is the optimal time point when one should stop dating more people and settle on a mate choice (and live with the decision)
* [https://metacpan.org/pod/Bio::SeqIO Perl Doc 2]
* Your best strategy is to date an initial sample of N individuals, rejecting all, and marry the next one ranked higher than any of your N individuals. The question is what is the optimal number for N.
* [https://metacpan.org/pod/Bio::Seq#trunc Perl Doc 3]
# The problem could be investigated by simulating a pool of 10 individuals, ranked from 1-10 (most desirable being 1) and then take a sample of N
* [http://bioperl.open-bio.org/wiki/HOWTO:SeqIO Perl Doc 4]
# You may only date one individual at a time
 
# You cannot go back to reach previously rejected candidates
* Practice Tests
# Simulate N from 0 to 9 (zero means marrying the first date, a sample size of zero)
# Browse & Filter "Big Data" files with UNIX filters
# For each N, obtain the probability of finding the perfect mate (i.e., ranked 1st) by running simulation 1000 times
## Genome file
# Plot barplot of probability versus sample size N.
### List all FASTA files in the "../../bio425/data" directory: <code>ls ../../bio425/data/*.fas</code>
# Expected answer: N=4
### Count number of sequences in a FASTA file: "../../bio425/data/GBB.pep": <code>grep -c ">" ../../bio425/data/GBB.pep</code>
### Count number of sequences in all FASTA file in this directory: <code>grep -c ">" ../../bio425/data/*.fas</code>
### Remove directory names from the above output: <code>grep ">" ../../bio425/data/*.fas | sed 's/^.\+data\///'</code> or <code>grep ">" ../../bio425/data/*.fas| cut -f5 -d'/'</code>
## GenBank file
### Show top 10 lines in "../../bio425/data/mdm2.gb": <code>head ../../bio425/data/mdm2.gb</code>
### Show bottom 10 lines: <code>tail ../../bio425/data/mdm2.gb</code>
### Extract all lines containing nucleotides: <code>cat ../../bio425/data/mdm2.gb | grep -P "^\s+\d+[atcg\s]+$"</code>
## Transcriptome file: a microarray data set
### Count the number of lines in "../../bio425/data/ge.dat": <code>wc -l ../../bio425/data/ge.dat</code>
### Count the number of genes: <code>cut -f1 ../../bio425/data/ge.dat | grep -vc "Desc"</code>
### Count the number of cells: <code>head -1 ../../bio425/data/ge.dat | tr '\t' '\n' | grep -vc "Desc"</code>
## Transcriptome file: an RNA-SEQ output file
### Count the nubmer of lines in "../../bio425/data/gene_exp.diff": <code>wc -l ../../bio425/data/gene_exp.diff</code>
### Show head: <code>head ../../bio425/data/gene_exp.diff</code>
### Show tail: <code>tail ../../bio425/data/gene_exp.diff</code>
### Count nubmer of "OK" gene pairs (valid comparisons): <code>grep -c "OK" ../../bio425/data/gene_exp.diff</code>
### Count nubmer of significantly different genes: <code>grep -c "yes$" ../../bio425/data/gene_exp.diff</code>
### Sort by "log2(fold_change)": <code>grep "yes$" ../../bio425/data/gene_exp.diff | cut -f1,10 | sort -k2 -n</code>
## A proteomics dataset: SILAC
### Count the number of line in "../../bio425/data/Jill-silac-batch-1.dat": <code>wc -l ../../bio425/data/Jill-silac-batch-1.dat</code>
### Show top: <code>head ../../bio425/data/Jill-silac-batch-1.dat</code>
### Show bottom: <code>wc -l ../../bio425/data/Jill-silac-batch-1.dat</code>
### Show results for P53 genes: <code>grep -w "TP53" ../../bio425/data/Jill-silac-batch-1.dat</code>
### Sort by "log_a_b_ratio": <code>sort -k5 -n ../../bio425/data/Jill-silac-batch-1.dat</code>
### Show ranking of P53: <code>sort -k5 -n -r ../../bio425/data/Jill-silac-batch-1.dat | cat -n | grep -w "TP53"</code>
 
 
Important information for solving the homework !
[https://www.dropbox.com/sh/fdy60u8nlzgim8f/AACm6vX2hdhg-JGi7ZeZUyKCa?dl=0 BioPerl Lecture videos]
 
 
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
! Assignment #6 - (<font color="red">Due Wednesday 3/23/2016, at 10am - Submit Responses on Blackboard</font>)
|- style="background-color:powder blue;"
| '''Unix Text Filters''' (5 pts) Show both commands and outputs for the following questions:<br>
# Identify homologs and build a phylogeny
## Copy genome file: "../../bio425/data/GBB.1con-splitted/Borrelia_burgdorferi_4075_lp54_plasmid_A.fas" to your home directory as "lp54.fas": <code>cp ../../bio425/data/GBB.1con-splitted/Borrelia_burgdorferi_4075_lp54_plasmid_A.fas lp54.fas</code>
## Identify ORFs in file using "long-orfs": <code>long-orfs lp54.fas lp54.coord</code>
## Extract nucleotide sequences: <code>extract lp54.fas lp54.coord > lp54.nuc</code>
## Use "bioseq" to translate into peptides, into "lp54.pep": <code>bioseq -t1 lp54.nuc > lp54.pep</code>
## Make a blast database: <code>makeblastdb -in lp54.pep -dbtype 'prot' -parse_seqids -out lp54</code>
## Copy file "../../bio425/data/BBA68.pep" to your home directory: <code>cp ../../bio425/data/BBA68.pep .</code>
## Find homologs of "BBA68.pep" in "lp54.pep" using "blastp": <code>blastp -query BBA68.pep -db lp54 -evalue 1e-5 -outfmt 6 | cut -f2 > BBA68.homologs</code>
## Extract all blast homologs using "blastdbcmd": <code>blastdbcmd -db lp54 -entry_batch BBA68.homologs > BBA68.homologs.pep</code>
## Align all homologs using "muscle": <code>muscle -in BBA68.homologs.pep -out BBA68.homologs.aligned</code>
## Build a tree of BBA65 homologs using "FastTree": <code>FastTree  BBA68.homologs.aligned > BBA68.dnd</code>
## Show tree with [http://www.evolgenius.info/evolview/ evolview]
# Review BASH "while" & "for" loops (see Assignment #3)
# Review the use Bio::SeqIO to read FASTA files and the use Bio::Seq methods to manipulate sequences (see Assignment #5)
 
'''Perl exercises (5 pts)'''
*Write an "extract.pl" using Perl & BioPerl that extract coding sequences from "coord" files. The script will use Bio::SeqIO to read the genome FASTA file ("mystery_seq1.fas", see Assignment #2 above). It will also read the "mystery_seq1.coord" (see Assignment #2 above) as the 2nd argument. Use Bio::Seq to obtain coding sequences and translate sequences. Your output of protein sequences should not contain stop codons except as the last codon. The following template helps you get started:
 
<syntaxhighlight lang=perl line enclose="div">
!/usr/bin/perl
use strict;
use warnings;
use lib '/data/biocs/b/bio425/bioperl-live';
use Bio::SeqIO;
# ----------------------------------------
# File            : extract.pl
# Author          : WGQ
# Date            : March 5, 2015
# Description    : Emulate glimmer EXTRACT program
# Input          : A FASTA file with 1 DNA seq and coord file from LONG-ORF
# Output          : A FASTA file with translated protein sequences
# ----------------------------------------
die "Usage: $0 <FASTA_file> <coord_file>\n" unless @ARGV > 0;
my ($fasta_file, $coord_file) = @ARGV;
my $fasta_input = Bio::SeqIO->new(<your code>); # create a file handle to read sequences from a file
my $output = Bio::SeqIO->new(-file=>">$fasta_file".".out", -format=>'fasta'); # create a file handle to output sequences into a file
my $seq_obj = $input->next_seq(); # get sequence object from FASTA file
# Read COORD file & extract sequences
open COORD, "<" . $coord_file;
while (<COORD>) {
      my $line = $_;
      chomp $line;
      next unless $line =~ /^\d+/; # skip lines except
      my ($seq_id, $cor1, $cor2, $strand, $score) = split /\s+/, $line; # split line on white spaces
      if (<your code: if strand is positive>) {
        <your code: use trunc function to get sub-sequence as an object>
      } else {
        <your code: use the trunc() function to get sub-sequence as an object>
        <your code: use the revcom() function to reverse-complement the seq object>
      }
      <your code: use translate() function to obtain protein sequence as an abject, "$pro_obj">
      $output->write_seq($pro_obj);
}
close COORD;
exit;
</syntaxhighlight>
 
 
* The following Perl code dumps nucleotide and codon frequencies given a coding sequence. It is based on the [http://search.cpan.org/~cjfields/BioPerl-1.6.923/Bio/Tools/SeqStats.pm Bio::Tools::SeqStats module of BioPerl]. Copy and run it with <code>./seq-stats.pl ../../bio425/data/B31_ospA.fas</code> to understand its behavior.
<syntaxhighlight lang=perl line enclose="div">
#!/usr/bin/perl -w
use strict;
use lib '/data/biocs/b/bio425/bioperl-live';
use Bio::Tools::SeqStats;
use Bio::SeqIO;
use Data::Dumper;
die "Usage: $0 <fasta>\n" unless @ARGV == 1;
my $filename = shift @ARGV;
my $in = Bio::SeqIO->new(-file=>$filename, -format=>'fasta');
my $seqobj = $in->next_seq();
my $seq_stats = Bio::Tools::SeqStats->new($seqobj);
my $monomers = $seq_stats->count_monomers();
my $codons = $seq_stats->count_codons();
print Dumper($monomers, $codons);
exit;
</syntaxhighlight>
* Write a Perl code ("my-seq-stats.pl") to emulate its two methods ("count_monomers" & "count_codons") with the following specific requirements:
** Read the fasta file with Bio::SeqIO
** Write a subroutine named as "count_monomers" to count and print the frequency of each nucleotide (in percentage), sorted by nucleotide type. The argument for the subroutine would be the sequence as a string, as in <code>count_monomers($seqobj->seq)</code>. The return value would be a reference to a hash (the same as the <code>$monomers</code> in the above code).
** Hint: use hash to count unique bases, see [http://my.safaribooksonline.com/book/programming/perl/1565922433/arrays/ch04-13496#X2ludGVybmFsX0h0bWxWaWV3P3htbGlkPTEtNTY1OTItMjQzLTMlMkZjaDA0LTE3NDIxJnF1ZXJ5PQ== Perl Cookbook example], especially the code listed under "4.6.2.4. Faster but different"
* In a similar way, write a subroutine named as "count_codons" to count and print the frequency of each ''codon'', sorted by codons. Your codon counts should be the same as those produced by the listed code.
|}
 
===March 23 (No Class; College is on a Friday schedule)===
===March 30. Midterm Q&A (Midterm will be posted March 26th) ===
 
Class meets at the regular time & room, but no new lecture. Instead, bring all the questions
you have for the material we covered so far - we will answer all questions so everyone is well
prepared for completing the midterm.
 
{| class="wikitable sortable mw-collapsible"
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
|- style="background-color:lightsteelblue;"
! Assignment #7 (<font color="red">MIDTERM: Due Wednesday 4/6/2016, at 10am - Submit Responses on Blackboard</font>)
! Submitted Codes
|- style="background-color:powderblue;"
|- style="background-color:powderblue;"
 
|
* (5 pts)  Browse & Filter "Big Data" files with UNIX filters
* By Weigang
Genome file
<syntaxhighlight lang="bash">
List all FASTA files in the "../../bio425/data" directory: <code>ls ../../bio425/data/*.fas</code>
pick.candidate <- function(min, array) {
Count number of sequences in a FASTA file: "../../bio425/data/GBB.pep": <code>grep -c ">" ../../bio425/data/GBB.pep</code>
  for (i in 1:length(array)) {
### Count number of sequences in all FASTA file in this directory: <code>grep -c ">" ../../bio425/data/*.fas</code>
    if (array[i] < min) {
### Remove directory names from the above output: <code>grep ">" ../../bio425/data/*.fas | sed 's/^.\+data\///'</code> or <code>grep ">" ../../bio425/data/*.fas| cut -f5 -d'/'</code>
      return(array[i])
## GenBank file
    } else {next}
### Show top 10 lines in "../../bio425/data/mdm2.gb": <code>head ../../bio425/data/mdm2.gb</code>
  }
### Show bottom 10 lines: <code>tail ../../bio425/data/mdm2.gb</code>
  return(0) # No 1. has been sampled and rejected
### Extract all lines containing nucleotides: <code>cat ../../bio425/data/mdm2.gb | grep -P "^\s+\d+[atcg\s]+$"</code>
## Transcriptome file: a microarray data set
### Count the number of lines in "../../bio425/data/ge.dat": <code>wc -l ../../bio425/data/ge.dat</code>
### Count the number of genes: <code>cut -f1 ../../bio425/data/ge.dat | grep -vc "Desc"</code>
### Count the number of cells: <code>head -1 ../../bio425/data/ge.dat | tr '\t' '\n' | grep -vc "Desc"</code>
## Transcriptome file: an RNA-SEQ output file
### Count the nubmer of lines in "../../bio425/data/gene_exp.diff": <code>wc -l ../../bio425/data/gene_exp.diff</code>
### Show head: <code>head ../../bio425/data/gene_exp.diff</code>
### Show tail: <code>tail ../../bio425/data/gene_exp.diff</code>
### Count nubmer of "OK" gene pairs (valid comparisons): <code>grep -c "OK" ../../bio425/data/gene_exp.diff</code>
### Count nubmer of significantly different genes: <code>grep -c "yes$" ../../bio425/data/gene_exp.diff</code>
### Sort by "log2(fold_change)": <code>grep "yes$" ../../bio425/data/gene_exp.diff | cut -f1,10 | sort -k2 -n</code>
## A proteomics dataset: SILAC
### Count the number of line in "../../bio425/data/Jill-silac-batch-1.dat": <code>wc -l ../../bio425/data/Jill-silac-batch-1.dat</code>
### Show top: <code>head ../../bio425/data/Jill-silac-batch-1.dat</code>
### Show bottom: <code>wc -l ../../bio425/data/Jill-silac-batch-1.dat</code>
### Show results for P53 genes: <code>grep -w "TP53" ../../bio425/data/Jill-silac-batch-1.dat</code>
### Sort by "log_a_b_ratio": <code>sort -k5 -n ../../bio425/data/Jill-silac-batch-1.dat</code>
### Show ranking of P53: <code>sort -k5 -n -r ../../bio425/data/Jill-silac-batch-1.dat | cat -n | grep -w "TP53"</code>
 
* (5 pts) Test the "GT-AG" rule of introns. Complete the following script to extract all intron sequences from this file <code>../../bio425/data/mdm2.gb</code>.
** After carefully studying and deciphering the script, replace "?"s (a total of 6) in the script with real numbers
** Save the script as "splice.pl" & run with the four options individually, as in <code>splice -i mdm2.gb</code>.
** Make a [http://weblogo.berkeley.edu/ SeqLogo] to show that all introns start with "GT" and ends with "AG". To do so, copy & paste individual sequences at the donor sites [http://weblogo.threeplusone.com/create.cgi into this text box] and click "Create Logo". Print and hand in the resulting image file. Indicate which sites are conserved.
** Repeat for the acceptor-site sequences. Indicate which sites are conserved.
<syntaxhighlight lang=perl line enclose="div">
#!/usr/bin/perl -w
 
use strict;
use lib '/data/biocs/b/bio425/bioperl-live';
use Bio::SeqIO;
use Data::Dumper;
use Getopt::Std;
# ----------------------------------------
# Author          : WGQ
# Date            : April 2, 2015
# Description    : Extract intron, exon, and junction sequences
# Input          : A GenBank file containing introns
# Output          : A Fasta file to feed into WebLogo
# Gene Model      : ---exon[i]-donor[i]-intron[i]-acceptor[i]-exon[i+1]-donor[i+1]-intron[i+1]-acceptor[i+1]---
# ----------------------------------------
# Part 1. Read 4 options & genbank file
my %opts;
getopts("eida", \%opts); # process options
die "Usage: $0 [-e(xon) -i(ntron) --d(onor) --a(cceptor)] <a GenBank file>\n" unless @ARGV == 1 && ($opts{e} || $opts{i} || $opts{d} || $opts{a}); # print usage if no input file or no options given
my $gb = shift @ARGV;
my $in = Bio::SeqIO->new(-file=>$gb, -format=>'genbank');
my $seq = $in->next_seq();
my (@exons, @introns, @donors, @acceptors); # declare arrays as data contains
 
# Part 2. Extract exon information
foreach my $feat ( $seq->get_SeqFeatures() ) { # loop through each genbank feature
    next unless $feat->primary_tag() eq "mRNA"; # skip unless the feature is tagged as "mRNA"
    my $location_obj = $feat->location(); # splice coordinates saved as a Bio::Location::Split object
    my $exon_id = 1;
    foreach my $loc ($location_obj->each_Location) { # access each exon coords object
        push @exons, {
            'order' => $exon_id++,
            'start' => $loc->start,
            'end' => $loc->end
        };
    }
}
}
 
candidates <- 1:10;
# print Dumper(\@exons); exit; # uncomment to see results of parsed exons
output <- sapply(0:9, function(x) {
 
  ct <- 0;
# Part 3. Calcuate intron, donor, acceptor objects
  for(k in 1:1000) {
## Extract introns coords:
    if (x==0) { # no sample, marry the 1st guy
for (my $i=0; $i<$#exons; $i++) { # for each exon
      sampled <- sample(candidates, 1);
    push @introns, {
      if (sampled == 1) {ct <- ct+1}
        'order' => $exons[$i]->{order},
     } else {
        'start' => $exons[$i]->{end}+1, # intron i starts at (exon i)'s end + 1
      sampled <- sample(candidates, x);
        'end' => $exons[$i+1]->{start}-1, # intron i ends at (exon i+1)'s start -1
      not.sampled <- candidates[-sampled];
    };
      not.sampled <- sample(not.sampled);
}
      if (pick.candidate(min = min(sampled), array = not.sampled) == 1) {ct <- ct+1}
 
# print Dumper(\@introns); exit; # uncomment to see introns
 
## Donor sites (-10 to -1 of an exon and 1-20 of intron)
for (my $i=0; $i<$#exons; $i++) {
    push @donors, {
        'order' => $exons[$i]->{order},
        'start' => $exons[$i]->{end} - ?, # donor i starts at (exon i)'s -10 to -1
        'end' => $introns[$i]->{start} + ?, # donor i ends at (intron i+1)'s 1 to 20
    };
}
 
# print Dumper(\@donors); exit; # uncomment to see donor sequences
 
## Acceptor sites (-20 to -1 of an intron and 1-10 of exon)
for (my $i=0; $i<$#exons; $i++) {
    push @acceptors, {
        'order' => $exons[$i]->{order},
        'start' => $introns[$i]->{end} - ?, # acceptor i starts at (intron i)'s -20 to -1
        'end' => $exons[$i+1]->{start} + ?, # acceptor i ends at (exon i+1)'s 1 to 10
     };
}
 
# print Dumper(\@acceptors); exit; # uncomment to see acceptor sequences
 
# Part 4. Output sequences
&print_seq("intron", \@introns) if $opts{i};
&print_seq("exon", \@exons) if $opts{e};
&print_seq("donor", \@donors) if $opts{d};
&print_seq("acceptor", \@acceptors) if $opts{a};
 
exit;
 
sub print_seq {
    my ($tag, $ref) = @_;
    my @bins = @$ref;
    foreach my $bin (@bins) {
        my $out = Bio::SeqIO->new(-format=>'fasta');
        my $seq =$seq->trunc(?, ?);
        $seq->id($tag . "_" . $bin->{order});
        $out->write_seq($seq);
     }
     }
}
  }
  return(ct);
})
barplot(output/1e3, names.arg = 0:9, xlab = "number of sampled dates", las=1, main = "Optimal stopping for dating (N=10 candidates)", ylab = "Prob(marrying No.1)")
</syntaxhighlight>
</syntaxhighlight>
|}
|}
 
==(Previous week) Feb 3, 2017 "US Presidents" (Due 2/10/2017)==
===April 6: Transcriptome (1): Univariate statistics===
[[File:Sim-presidents.png|thumbnail]]
* Learning goals: 1. Microarray technology; 2. Introduction to R
* Download [[File:Presidents.txt|thumbnail]]: 1st column is the order, 2nd column is the name, the 3rd column is the year of inauguration; tab-separated
* Lecture Slides: [[File:Session-7-small.pdf|thumbnail]]
* Your job is to create an R, Perl, or Python script called “us-presidents”, which will
* R Tutorial 1
# Read the table
# Set a working directory
# Store the original/correct order
## Using a Linux terminal, make a directory called <code>R-work</code>, "cd" into this directory.
# Shuffle/permute the rows and record the new order
## Start R studio: <code>Type "rstudio &" on terminal prompt and hit return</code> ('''Note: This link works only if you use computers in the 1000G lab. Install and use your own R & R Studio for homework assignments''')
# Count the number of matching orders
## Find working directory with <code>getwd()</code>
# Repeat Steps 3-4 for a 1000 times
# The basic R data structure: Vector
# Plot histogram or barplot (better) to show distribution of matching counts
## Construct a character vector with <code>c.vect=c("red", "blue", "green", "orange", "yellow", "cyan")</code>
# Hint: For R, use the sample() function. For Perl, use the rand() function.
## Construct a numerical vector with <code>n.vect=c(2, 3, 6, 10, 1, 0, 1, 1, 10, 30)</code>
## Construct vectors using functions
###<code>n.seq1=1:20</code>
###<code>n.seq2=seq(from=1, to=20, by=2)</code>
###<code>n.rep1=rep(1:4, times=2)</code>
###<code>n.rep2=rep(1:4, times=2, each=2)</code>
## Use '''built-in help''' of R functions: <code>?seq</code> or <code>help(seq)</code>
## Find out data class using the '''class''' function: <code>class(c.vect)</code>
# Access vector elements
## A single value: <code>n.vect[2]</code>
## A subset: <code>n.vect[3:6]</code>
## An arbitrary subset: <code>n.vect[c(3,7)]</code>
## Exclusion: <code>n.vect[-8]</code>
## Conditional: <code>n.vect[n.vect>5]</code>
# Apply functions
## Length: <code>length(n.vect)</code>
## Range: <code>range(n.vect); min(n.vect); max(n.vect)</code>
## Descriptive statistics: <code>sum(n.vect); var(n.vect); sd(n.vect)</code>
## Sort: <code>order(n.vect)</code>. How would you get an ordered vector of n.vect? [Hint: use <code>?order</code> to find the return values]
## Arithmetic: <code>n.vect +10; n.vect *10; n.vect / 10</code>
# Quit an R session
## Click on the "History" tab to review all your commands
## Save your commands into a file by opening a new "R Script" and save it as <code>vector.R</code>
## Save all your data to a default file '''.RData''' and all your commands to a default file ".Rhistory" with <code>save.image()</code>
## Quit R-Studio with <code>q()</code>
* R tutorial 2. Data distribution using histogram
# Start a new R studio session and set working directory to <code>~/R-work</code>
# Generate a vector of 100 normal deviates using <code>x.ran=rnorm(100)</code>
# Visualize the data distribution using <code>hist(x.ran)</code>
# Explore help file using <code>?help; example(hist)</code>
# How to break into smaller bins? How to add color? How to change x- and y-axis labels? Change the main title?
# Test if the data points are normally distributed.[Hints: use qqnorm() and qqline()]
# Save a copy of your plot using "Export"->"Save Plot as PDF"
* R tutorial 3. Matrix & Data Frames
# Using a Linux terminal, make a copy of breast-cancer transcriptome file <code>/data/biocs/b/bio425/data/ge.dat</code> in your R working directory
# Start a new R studio session and set working directory to <code>~/R-work</code>
# Read the transcriptome file using <code>ge=read.table("ge.dat", header=T, row.names=1, sep="\t")</code>
## What is the data class of <code>ge</code>?
## Access data frame. What do the following commands return? <code>ge[1,]; ge[1:5,]; ge[,1]; ge[,1:5]; ge[1:5, 1:5]</code>
## Apply functions: <code>head(ge); tail(ge); dim(ge)</code>
# Save a copy of an object: <code>ge.df=ge</code>.
# Transform the transcriptome into a numerical matrix for subsequent operations: <code>ge=as.matrix(ge)</code>
#  Test if the expression values are normally distributed.
# Save and quit the R session
* In class challenge: Replicate normalization in Figure 4.8 (pg 208)
{| class="wikitable sortable mw-collapsible"
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
|- style="background-color:lightsteelblue;"
! Assignment #7 (<font color="red">Due Wednesday 4/6/2016, at 10am - Submit Responses on Blackboard</font>)
! Submitted Codes
|- style="background-color:powderblue;"
|- style="background-color:powderblue;"
|
|  
* (5 pts) Test the "GT-AG" rule of introns. Complete the following script to extract all intron sequences from this file <code>../../bio425/data/mdm2.gb</code>.
* By Mei
** After carefully studying and deciphering the script, replace "?"s (a total of 6) in the script with real numbers
<syntaxhighlight lang="bash">
** Save the script as "splice.pl" & run with the four options individually, as in <code>splice -i mdm2.gb</code>.
pres.list <- lapply(1:1000, function(x) pres[sample(nrow(pres)),])
** Make a [http://weblogo.berkeley.edu/ SeqLogo] to show that all introns start with "GT" and ends with "AG". To do so, copy & paste individual sequences at the donor sites [http://weblogo.threeplusone.com/create.cgi into this text box] and click "Create Logo". Print and hand in the resulting image file. Indicate which sites are conserved.
cts <- sapply(pres.list, function(x) {
** Repeat for the acceptor-site sequences. Indicate which sites are conserved.
  ct.match <- 0;
<syntaxhighlight lang=perl line enclose="div">
  for (i in 1:45){
#!/usr/bin/perl -w
     if (pres$order[i] == x[i,1]){
 
      #cat(as.character(x[i,2]), x[i,1], "\n") #as.character to avoid the factor info
use strict;
      ct.match <- ct.match + 1;
use lib '/data/biocs/b/bio425/bioperl-live';
      #cat(ct.match)
use Bio::SeqIO;
use Data::Dumper;
use Getopt::Std;
# ----------------------------------------
# Author          : WGQ
# Date            : April 2, 2015
# Description     : Extract intron, exon, and junction sequences
# Input          : A GenBank file containing introns
# Output          : A Fasta file to feed into WebLogo
# Gene Model      : ---exon[i]-donor[i]-intron[i]-acceptor[i]-exon[i+1]-donor[i+1]-intron[i+1]-acceptor[i+1]---
# ----------------------------------------
# Part 1. Read 4 options & genbank file
my %opts;
getopts("eida", \%opts); # process options
die "Usage: $0 [-e(xon) -i(ntron) --d(onor) --a(cceptor)] <a GenBank file>\n" unless @ARGV == 1 && ($opts{e} || $opts{i} || $opts{d} || $opts{a}); # print usage if no input file or no options given
my $gb = shift @ARGV;
my $in = Bio::SeqIO->new(-file=>$gb, -format=>'genbank');
my $seq = $in->next_seq();
my (@exons, @introns, @donors, @acceptors); # declare arrays as data contains
 
# Part 2. Extract exon information
foreach my $feat ( $seq->get_SeqFeatures() ) { # loop through each genbank feature
    next unless $feat->primary_tag() eq "mRNA"; # skip unless the feature is tagged as "mRNA"
    my $location_obj = $feat->location(); # splice coordinates saved as a Bio::Location::Split object
    my $exon_id = 1;
    foreach my $loc ($location_obj->each_Location) { # access each exon coords object
        push @exons, {
            'order' => $exon_id++,
            'start' => $loc->start,
            'end' => $loc->end
        };
     }
     }
}
  }
 
  ct.match #return
# print Dumper(\@exons); exit; # uncomment to see results of parsed exons
})
 
barplot(table(cts),xlab = "Number of matches per shuffle", border = "hotpink", col = "pink", ylab = paste("Frequency total of:",length(pres.list)), main = "US Presidents")
# Part 3. Calcuate intron, donor, acceptor objects
## Extract introns coords:
for (my $i=0; $i<$#exons; $i++) { # for each exon
    push @introns, {
        'order' => $exons[$i]->{order},
        'start' => $exons[$i]->{end}+1, # intron i starts at (exon i)'s end + 1
        'end' => $exons[$i+1]->{start}-1, # intron i ends at (exon i+1)'s start -1
    };
}
 
# print Dumper(\@introns); exit; # uncomment to see introns
 
## Donor sites (-10 to -1 of an exon and 1-20 of intron)
for (my $i=0; $i<$#exons; $i++) {
    push @donors, {
        'order' => $exons[$i]->{order},
        'start' => $exons[$i]->{end} - ?, # donor i starts at (exon i)'s -10 to -1
        'end' => $introns[$i]->{start} + ?, # donor i ends at (intron i+1)'s 1 to 20
    };
}
 
# print Dumper(\@donors); exit; # uncomment to see donor sequences
 
## Acceptor sites (-20 to -1 of an intron and 1-10 of exon)
for (my $i=0; $i<$#exons; $i++) {
    push @acceptors, {
        'order' => $exons[$i]->{order},
        'start' => $introns[$i]->{end} - ?, # acceptor i starts at (intron i)'s -20 to -1
        'end' => $exons[$i+1]->{start} + ?, # acceptor i ends at (exon i+1)'s 1 to 10
    };
}
 
# print Dumper(\@acceptors); exit; # uncomment to see acceptor sequences
 
# Part 4. Output sequences
&print_seq("intron", \@introns) if $opts{i};
&print_seq("exon", \@exons) if $opts{e};
&print_seq("donor", \@donors) if $opts{d};
&print_seq("acceptor", \@acceptors) if $opts{a};
 
exit;
 
sub print_seq {
    my ($tag, $ref) = @_;
    my @bins = @$ref;
    foreach my $bin (@bins) {
        my $out = Bio::SeqIO->new(-format=>'fasta');
        my $seq =$seq->trunc(?, ?);
        $seq->id($tag . "_" . $bin->{order});
        $out->write_seq($seq);
    }
}
</syntaxhighlight>
</syntaxhighlight>
* (2 pts) Install [http://streaming.stat.iastate.edu/CRAN/ R (choose a binary distribution)] first and then [http://www.rstudio.com/ R Studio] to your home/laptop computer [No need to show any output. I trust you on this.]
* By John
* (4 pts) Make 2 histograms based on the "ge" data set: (a) expression values for a single (any one if fine) selected cell line (cell lines are in columns, using small-sized bins, e.g., "br=100") and (b) expression values for a selected (any one if fine) gene (genes are in rows). Show all R commands and the final plots (label the plot title with cell line or gene name) ['''Notes:''' You need to download a copy of the <code>ge.dat</code> onto your own computer. On Mac, you could run this command: <code>scp your_user_name@eniac.cs.hunter.cuny.edu:ge.dat  ge.dat</code>. On Windows, download and install <code>sftp</code> or<code> WinSCP</code> for file transfer from/to your eniac account.]
<syntaxhighlight lang="python">
* (4 pts) Use the help() and example() functions to learn how to use the which() and grep() functions. Show how to use these two functions, respectively, to display gene expression values for a gene called "ESR1" in the "ge" data set. [Hint: use the rownames() function to obtain gene names of the "ge" data set.]
import pandas as pd
|}
import numpy as np
from pandas import DataFrame
import matplotlib.pyplot as plt


===April 6: Transcriptome (2): Bi- and Multi-variate statistics===
df = pd.read_table("presidents.txt", names=["num", "name", "presidency"])
* Learning goal: (1) multivariate clustering analysis; (2) Classification of breast-cancer subtypes
* Lecture slides: [[File:Session-8-small.pdf|thumbnail]]
* Tutorial 1. Gene filtering
# Open a new R Studio session and set working directory to R-work
# Load the default workspace and plot a histogram of the gene expression using <code>hist(ge, br=100)</code>. Answer the following questions:
## Make sure <code>ge</code> is a matrix class. If not, review the last session on how to make one
## What is the range of gene expression values? Minimum? Maximum? <pre style="white-space: pre-wrap;">range(ge); min(ge); max(ge)</pre>
## Are these values normally distributed? Test using <code>qqnorm</code> and <code>qqline</code>.
## If not, which data range is over-represented? Why? <pre style="white-space: pre-wrap;">Low gene expression values are over-represented, because most genes are NOT expressed in a particular cell type/tissue.</pre>
# '''Discussion Questions:'''
## What does each number represent? <pre style="white-space: pre-wrap;">log2(cDNA)</pre>
## Why is there an over-representation of genes with low expression values? <pre style="white-space: pre-wrap;">Because most genes are not expressed in these cancer cells</pre>
## How to filter out these genes? <pre style="white-space: pre-wrap;">Remove genes that are expressed in low amount in these cells, by using the function pOverA (see below)</pre>
## How to test if the filtering is successful or not? <pre style="white-space: pre-wrap;">Run qqnorm() and qqline()</pre>
# Remove genes that do not vary significantly among breast-cancer cells
## Download the genefilter library from [http://bioconductor.org BioConductor] using the following code <pre style="white-space: pre-wrap;">source("http://bioconductor.org/biocLite.R"); biocLite("genefilter")</pre>
## Load the genefilter library with <code>library(genefilter)</code>
## Create a gene-filter function: <code>f1=pOverA(p=0.5, A=log2(100))</code>. What is the <code>pOverA</code> function? <pre style="white-space: pre-wrap;">Remove genes expressed with lower than log2(100) amount in half of the cells</pre>
## Obtain indices of genes significantly vary among the cells: <code>index.retained=genefilter(ge, f1)</code>
## Get filtered expression matrix: <code>ge.filtered=ge[index.retained, ]</code>. How many genes are left? <pre style="white-space: pre-wrap;">dim(ge.filtered)</pre>
# Test the normality of the filtered data set
## Plot with <code>hist(); qqnorm(); and qqline()</code>. Are the filtered data normally distributed?
## Plot with <code>boxplot(ge.filtered)</code>. Are expression levels distributed similarly among the cells?
* Tutorial 2. Select genes vary the most in gene expression
# Explore the function <code>mad</code>. What are the input and output? <pre style="white-space: pre-wrap;">Input: a numerical array. Output: median deviation</pre>
# Calculate mad for each gene: <code>mad.ge=apply(ge.filtered, MARGIN=1, FUN=mad)</code>
# Obtain indices of the top 100 most variable genes: <code>mad.top=order(mad.ge, decreasing=T)[1:100]</code>
# Obtain a matrix of most variable genes: <code>ge.var=ge.filter[mad.top,]</code>
* Part 3. Classify cancer subtypes based on gene expression levels
# '''Discussion Questions:'''
## How would you measure the difference between a one-dimensional variable? <pre style="white-space: pre-wrap;">d=|x1-x2|</pre>
## How would you measure the difference between a two-dimensional variable? <pre style="white-space: pre-wrap;">d=|x1-x2|+|y1-y2|</pre>
## How would you measure the Euclidean difference between a three-dimensional variable? <pre style="white-space: pre-wrap;">d=|x1-x2|+|y1-y2|+|z1-z2|</pre>
## How would you measure the Euclidean difference between a multi-dimensional variable? <pre style="white-space: pre-wrap;">d=SQRT(sum([xi-xj]^2))</pre>
## To measure similarities between cells based on their gene expression values, is it better to use Euclidean or correlation distances? <pre style="white-space: pre-wrap;">Correlational distance measures similarity in trends regardless of magnitude.</pre>
# Calculate correlation matrix between cells: <code>cor.cell=cor(ge.var)</code>
# Calculate correlation matrix between genes: <code>cor.gene=cor(t(ge.var))</code> What does the t() function do? <pre style="white-space: pre-wrap;">transposition (turn rows into columns and columns into rows)</pre>
# Obtain correlational distances between cells: <code>dg.cell=as.dendrogram(hclust(as.dist(1-cor.cell)))</code>
# Obtain correlational distances between genes: <code>dg.gene=as.dendrogram(hclust(as.dist(1-cor.gene)))</code>
# Plot a heat map: <code>heatmap(ge.var, Colv=dg.cell, Rowv=dg.gene)</code>
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
! Assignment #8 (<font color="red">Due Wednesday 4/13/2016, at 10am - Submit Responses on Blackboard</font>)
|- style="background-color:powderblue;"
|
(10 pts) The following table lists expression levels of a gene measured under an Experimental and a Control conditions, each of which was replicated five times:
{| class="wikitable"
|-
!  !! Experimental !! Control
|-
| Replicate 1 || 606 || 287
|-
| Replicate 2 || 441 || 198
|-
| Replicate 3 || 702 || 366
|-
| Replicate 4 || 597 || 255
|-
| Replicate 5 || 888 || 402
|}
Test significance of gene expressions between the Exp and Ref groups. Show all R commands, test results, & plots.
* Plot a boxplot
* Plot a vertical stripcharts. Why is a stripchart more informative than a boxplot?
* Perform a paired t-test, using the log2 scale. Are gene expression values significantly different between the Exp and Ref groups?
* Plot a scatter plot and add a regression line. Show results of regression analysis, include the R-squared value and the p value. Are the expression levels between the two conditions significantly correlated?
|}


===April 13: Transcriptome (3). RNA-SEQ Analysis===
name_list = list(df.name)
* Learning goal: RNA-SEQ pipeline
* Lecture slides: [[File:Session-9-small.pdf|thumbnail]]
* Read data source [http://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE32038 GSE32038]
* Tutorial 1. Visualize expression levels
<syntaxhighlight lang="bash" line enclose="div">
gene.fpkm=read.table("../../bio425/data/genes.fpkm_tracking", head=T, sep="\t")
dim(gene.fpkm)
head(gene.fpkm)
fpkm=gene.fpkm[which(gene.fpkm$C1_status=='OK' & gene.fpkm$C2_status=='OK'),] # filter low-quality values
hist(log10(fpkm$C1_FPKM), br=100)
plot(density(log10(fpkm$C1_FPKM)))
plot(log10(fpkm$C1_FPKM), log10(fpkm$C2_FPKM))
fpkm.m=log2(fpkm$C1_FPKM) + log2(fpkm$C2_FPKM)
fpkm.a=log2(fpkm$C1_FPKM) - log2(fpkm$C2_FPKM)
plot(fpkm.m, fpkm.a)
</syntaxhighlight>
* Tutorial 2. Test differential gene expressions
<syntaxhighlight lang="bash" line enclose="div">
gene=read.table("/data/biocs/b/bio425/data/gene_exp.diff", header=T, sep="\t")
dim(gene)
head(gene)
gene.ok=gene[which(gene$status=='OK'),] # filter
gene.sig=gene.ok[which(gene.ok$significant=="yes"),] # select significant genes
plot(gene.ok$log2.fold_change., -log10(gene.ok$q_value)) # volcano plot
plot(gene.ok$log2.fold_change., -log10(gene.ok$q_value), col=gene.ok$significant)
</syntaxhighlight>
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
! Assignment #9 (<font color="red">Due Wednesday 4/20/2016, at 10am - Submit Responses on Blackboard</font>)
|- style="background-color:powderblue;"
|
* (5 pts). Filter genes in the "ge.mat" using the "mad" function. Retain the top 50 most-variable genes. Calculate pairwise gene and cell correlation distances. Plot a heatmap. Answer the following questions:
** What does the "mad" function return? What is the biological rationale to use this function to filter genes, before identifying subtypes among cell lines?
** How many major subtypes of cancer cells and subgroups of genes you could identify, respectively?
** Label the genes as two largest groups (Group 1 & 2 Genes). Similarly, label the largest two groups of cell lines as "Group 1" and "Group 2" cells.  Explain the expression profiles of Group 1 and Group 2 genes in each subgroup of cells. (e.g., "Group 1 genes are up-regulated in Group 1 Cells and down-regulated in Group 2 Cells". Note that the "yellow" is high in heat color and "red" low).
* (5 pts). Transfer the two RNA-SEQ output files ("genes.fpkm_tracking" and "gene_exp.diff", both in the folder "bio425/data/") to your local computer (using "scp"). Make a histogram of the "C2" sample (use the first file), an MA plot to show differences between "C1" and "C2" in FPKM counts (use the first file), and a volcano plot (use the 2nd file). Show all your R commands. How many genes are significantly different between the two conditions (based on the 2nd file)?
|}


===April 20: Population Analysis (1)===
# create a list to store matches after each shuffle
* Learning goals: (1) SNP analysis (single-locus), (2) haplotype analysis (multiple loci)
shuffle_record = []
* Lecture slides: [[File:Session-10.pdf|thumbnail]]
Tutorial 1. Extract SNP sites
<syntaxhighlight lang="perl" line enclose="div">
#!/usr/bin/perl
# Author: WGQ
# Description: Examine each alignment site as a SNP or constant
# Input:  a DNA alignment
# Output: a haplotypes alignemnt
use strict;
use warnings;
use Data::Dumper;


# Part I. Read file and store NTs in a hash (use Bio::AlignIO to read an alignment is better)
# create a function in which the first argument is the original dataset; second argument is number of shuffles
my %seqs; # declare a hash as sequence container
def shuffler2(original, n):
my $length;
    record = []
while (<>) { # read file line by line
    for i in range(n):
  my $line = $_;
        num = 0
  chomp $line;
        each_shuffle = {}
  next unless $line =~ /^seq/; # skip lines unless it starts with "seq"
        temp = original.reindex(np.random.permutation(original.index)) # do shuffling for each
  my ($id, $seq) = split /\s+/, $line; # split on white spaces
        compare = original.num == temp.num
  $seqs{$id} = [ (split //, $seq) ]; # id as key, an array of nts as value
        matched_df = original.ix[compare[compare == True].index]
  $length = length($seq);
        for i in matched_df.index:
}
            each_shuffle[i] = matched_df.name[i]
        shuffle_record.append(each_shuffle)
        try:
            num = compare.value_counts()[1]
        except:
            pass
        record.append(num)
    return(record)
result2 = shuffler2(df, 1000)


my %is_snp; # declare a hash to store status of alignment columns
plt.hist(result2, color="yellow")
# Part II. Go through each site and report status
plt.title("Histgram for President Data")
for (my $pos = 0; $pos < $length; $pos++) {
plt.show()
  my %seen_nt; # declare a hash to get counts of each nt at a aligned position
</syntaxhighlight>  
  foreach my $id (keys %seqs) { # collect and count all nts at an aligned site
* By Weigang
    my $nt =  $seqs{$id}->[$pos];
<syntaxhighlight lang="bash">
    $seen_nt{$nt}++;
p <- read.table("Presidents.txt", sep="\t", header=F)
  }
colnames(p) <- c("order", "name", "inaug.year")
  my @key_nts = keys %seen_nt;
# Use "sapply" or "lapply" for loops: no need to pre-define a vector to store results
  if (@key_nts > 1) { # a SNP site has more than two nucelotdies
p.sim <- sapply(1:10000, function (x) {
    $is_snp{$pos} = 1;
   length(which(sample(p$order) == p$order))
   } else { # a constant site
   })
    $is_snp{$pos} = 0;
barplot(table(p.sim)/1e4, las=1)
   }
p.exp <- rpois(1e4, 1) # draw 10000 Poisson random deviates
}
mp <- barplot(table(p.exp)/1e4, las=1, xlab = "Num of matching presidents") # mid-point on x-axis
 
lines(mp[1:7], table(p.sim)/1e4, type="b", col=2) # add a line (Poisson-expected) to the barplot (simulated)
# Part III. Print haplotypes (i.e., nucleotides at SNP sites for each chromosome)
legend("topright", c("Simulated", "Poisson expectation"), col=1:2, lty=1)
foreach my $id (keys %seqs) { # for each chromosome
  print $id, "\t";
  for (my $pos = 0; $pos < $length; $pos++) { # for each site
    next unless $is_snp{$pos}; # skip constant site
    print $seqs{$id}->[$pos]; # print nt at a SNP site
  }
  print "\n";
}
 
exit;
</syntaxhighlight>
</syntaxhighlight>
Tutorial 2. Hardy-Weinberg Equilibrium: Calculate expected genotype frequencies
{| class="wikitable sortable mw-collapsible"
|- style="background-color:lightsteelblue;"
! Assignment #10 (<font color="red">Due Wednesday 4/27/2016, at 10am - Submit Responses on Blackboard</font>)
|- style="background-color:powderblue;"
|
* Based on the <code>../../bio425/data/ex3-1.aln</code>,
* (5 pts). Analyze alignment by eye: Define "allele", "SNP", and "haplotype". Print out the alignment
** Label all SNP sites
** Count frequency of each of the two alleles at the first SNP site
** List all haplotypes, and count frequency of each haplotype.
** How many SNP (i.e., variable) sites you expect to see in a window of DNA containing 10K bases among humans?
** How many nucleotide states (2, 3, or 4) you expect to see for an individual SNP site?
** Do you expect to see more or less transitons than tranvsersions?
** More synonymous or nonsynonymous SNPs (if the sequence codes for a protein)?
* (5 pts). Complete the above PERL script by replacing the two question marks. Compare your output to the haplotypes you obtained manually. They should be the same.
|}
|}


===May 4: Population Analysis (2)===
==(Future weeks) March 3, 2017 (Due March 10, 2017) Pi & E===
* Learning goals: Case-Control studies for gene mapping
* Lecture slides: [[File:Session-11-small.pdf|thumbnail]]
* Tutorial 1. Test linkage disequilibrium (LD)
* Tutorial 2. Genome-wide association studies (GWAS)
* Tutorial 3. Test of Hardy-Weinberg equilibrium (for random mating & presence of natural selection)
 
===May 11: Final Review===
* Review slides: [[File: Final-Review-small-bio425-sp-2015.pdf|thumbnail]]
* Tutorial 3. Database access with DBI
* [http://www.hunter.cuny.edu/te '''Teacher's evaluation''']
* All make-up assignments due May 20.
 
===May 18: Final Exam===
* Reminder: [http://www.hunter.cuny.edu/te '''Teacher's evaluation''']
 
==General Information==
===Course Description===
* '''Background:''' Biomedical research is becoming a high-throughput science. As a result, information technology plays an increasingly important role in biomedical discovery. Bioinformatics is a new interdisciplinary field formed between molecular biology and computer science.
* '''Contents:''' This course will introduce both bioinformatics '''theories''' and '''practices'''. Topics include: database searching, sequence alignment, molecular phylogenetics, structure prediction, and microarray analysis. The course is held in a UNIX-based instructional lab specifically configured for bioinformatics applications.
* '''Problem-based Learning (PBL):'''  For each session, students will work in groups to solve a set of bioinformatics problems. Instructor will serve as the facilitator rather than a lecturer. Evaluation of student performance include both active participation in the classroom work as well as quality of assignments (see [[#Grading Policy]]).
* '''Learning Goals:''' After competing the course, students should be able to perform most common bioinformatics analysis in a biomedical research setting. Specifically, students will be able to
** Approach biological questions evolutionarily ("Tree-thinking")
** Evaluate and interpret computational results statistically ("Statistical-thinking")
** Formulate informatics questions quantitatively and precisely ("Abstraction")
** Design efficient procedures to solve problems ("Algorithm-thinking")
** Manipulate high-volume textual data using UNIX tools, Perl/BioPerl, R, and Relational Database ("Data Visualization")
* '''Pre-requisites:''' This 3-credit course is designed for upper-level undergraduates and graduate students. Prior experiences in the UNIX Operating System and at least one programming language are required. Hunter pre-requisites are CSCI132 (Practical Unix and Perl Programming) and BIOL300 (Biochemistry) or BIOL302 (Molecular Genetics), or permission by the instructor. '''Warning: This is a programming-based bioinformatics course. Working knowledge of UNIX and Perl is required for successful completion of the course.'''
* '''Textbook:''' Gibson & Muse (2009). ''A Primer of Genome Science'' (Third Edition). Sinauer Associates, Inc.
* '''Academic Honesty:''' Hunter College regards acts of academic dishonesty (e.g., plagiarism, cheating on examinations, obtaining unfair advantage, and falsification of records and official documents) as serious offenses against the values of intellectual honesty. The College is committed to enforcing the CUNY Policy on Academic Integrity and will pursue cases of academic dishonesty according to the Hunter College Academic Integrity Procedures.
 
===Grading Policy===
* '''Treat assignments as take-home exams.''' Student performance will be evaluated by weekly assignments and projects. While these are take-home projects and students are allowed to work in groups and answers to some of the questions are provided in the back of the textbook, students are expected to compose the final short answers, computer commands, and code independently. There are virtually an unlimited number of ways to solve a computational problem, as are ways and personal styles to implement an algorithm. Writings and blocks of codes that are virtually exact copies between individual students will be investigated as possible cases of plagiarism (e.g., copies from the Internet, text book, or each other). In such a case, the instructor will hold closed-door exams for involved individuals. Zero credits will be given to ALL involved individuals if the instructor considers there is enough evidence for plagiarism. To avoid being investigated for plagiarism, <u>Do NOT copy from others or let others copy your work</u>.
* '''Submit assignments on the Blackboard''' Email attachments will NOT be accepted. Assignments will not be allowed past the due date and time and will be graded as zero. Each assignment will be graded based on timeliness (10%), whether executable or having major errors (50%), algorithm efficiency (10%), and readability in programming styles (30%, see [[#Assignment Expectations]]).
* '''The grading scheme'''
** Assignments (100 pts): 10 exercises.
** Mid-term (50 pts).
** Final Project (50 pts)
** Classroom performance (50 pts): Active engagement in classroom exercises and discussions
** Attendance (50 pts): 1 unexcused absences = 40; 2 absences = 30; More than 2 = 0.
 
===Assignment Expectations===
* Use a programming editor (e.g., vi, emacs, sublime) so you could have features like automatic syntax highlighting, indentation, and matching of quotes and parenthesis.
* All PERL code must begin with "use strict; and use warnings;" statements. For each assignment, unless otherwise stated, I would like the '''full text''' of the source code. Since you cannot print using the text editor in the lab (even if you are connected from home), you must copy and paste the code into a word processor or a local text editor. If you are using a word processor, '''change the font to a fixed-width/monospace font'''. On Windows, this is usually '''Courier'''.
* Also, unless otherwise stated, both the '''input and the output of the program must be submitted as well'''. This should '''also be in fixed-width font''', and you should '''label''' it in such a way so that I know it is the program's input/output. This is so that I know that you've run the program, what data you have used, and what the program produced. If you are working from the lab, one option is to email the code to yourself, change the font, and then print it somewhere else as there is no printer in the lab.
* <u>[[Media: template-basic.png|Recommended Style]]</u>
* <u>[[Media: bad-submission.png| Bad Style]]</u>
 
==Useful Links==
 
===Unix Tutorials===
* Oreilly Book for the virtues of command-line tools: [http://www.oreilly.com/pub/e/3115 Data Science at Command Line] by Jeroen Janssens
* A very nice [http://www.ee.surrey.ac.uk/Teaching/Unix/ UNIX tutorial] (you will only need up to, and including, tutorial 4).
* FOSSWire's [http://files.fosswire.com/2007/08/fwunixref.pdf Unix/Linux command reference] (PDF). Of use to you: "File commands", "SSH", "Searching" and "Shortcuts".
 
===Perl Help===
* Professor Stewart Weiss has taught CSCI132, a UNIX and Perl class. His slides go into much greater detail and are an invaluable resource. They can be found on his course page [http://compsci.hunter.cuny.edu/~sweiss/course_materials/csci132/csci132_f10.php here].
* Perl documentation at [http://perldoc.perl.org perldoc.perl.org]. Besides that, running the perldoc command before either a function (with the -f option ie, perldoc -f substr) or a perl module (ie, perldoc Bio::Seq) can get you similar results without having to leave the terminal.
 
===Regular Expression===
* [http://www.cheatography.com/davechild/cheat-sheets/regular-expressions/ A regular expression cheatsheet]
 
===Bioperl===
* BioPerl's [http://www.bioperl.org/wiki/HOWTOs HOWTOs page].
* BioPerl-live [http://doc.bioperl.org/bioperl-live developer documentation]. (We use bioperl-live in class.)
* Yozen's tutorial on [http://diverge.hunter.cuny.edu/wiki/HOWTO:Bioperl-live_on_Mac_OS_X installing bioperl-live on your own Mac OS X machine]. (Let me know if there are any issues!).
* [https://spreadsheets.google.com/pub?key=0AjfPzjrqY7BndHpyRHlDZUlGcktINm1IbXVzX1QzMXc&single=true&gid=0&output=html A small table] showing some methods for BioPerl modules with usage and return values.
 
===SQL===
* [https://docs.google.com/document/d/1zYLPeenwsqPYchkpXnndzphBbTKqX2GjjLHDxlBnt78/edit?hl=en&authkey=CLnh_88K SQL Primer], written by Yozen.

Revision as of 20:43, 25 February 2017

(This week) Feb 24, 2017 "Idiots" (Due 3/3/2017)

  • Source: Paul Nahin (2000), "Dueling Idiots". Problem #2: "When Idiots Duel"
  • Game: Idiots A and B decide to duel with a gun in the following way: They will insert a single bullet into the gun's cylinder. The cylinder has a total of 6 slots. Idiot A will spin the cylinder and shoot at B. If the gun doesn't fire, then A will give the gun to B, who will spin the cylinder and then shoot at A. This back-and-forth duel will continue until one fool shoots (and kills) the other.
  • Questions: (1) what is the probability that A will win (and B dies); (2) What is the average number of trigger pulls before someone dies?

(Last week) Feb 17, 2017 "Birthday" (Due 2/24/2017)

B-day.png
  • Problem: What is the probability NONE of the N people in a room sharing a birthday?
  1. Randomly select N individuals and record their B-days
  2. Count the B-days NOT shared by ANY two individuals
  3. Repeat (for each N) 1000 times, and obtain probability by averaging the previous counts (i.e., divided by 1000)
  4. Vary N from 10 to 100, increment by 10
  5. Plot probability of no-shared B-Day (Y-axis) versus N (x-axis), with either a stripchart or boxplot, or both
Submitted Codes
  • By weigang
days <- 1:365;

find.overlap <- function(x) { return(length(which(table(x)>1))) }

output <- sapply(1:100, function(x) { # num of people in the room
  ct.no.overlap <- 0;
  for (k in 1:100) {
    bdays <- sample(days, x, replace = T);
    ct <- find.overlap(bdays);
    if (!ct) { ct.no.overlap <- ct.no.overlap + 1}
  }
  return(ct.no.overlap);
})
 
plot(1:100, output/100, xlab="Group size", ylab = "Prob (no shared b-day)", las=1, main="B-day (sim=100 times)", type="l")
abline(h=0.5, lwd=2, col=2, lty=2)
abline(v=seq(0, 100, 5), col="gray")

(Previous week) Feb 10, 2017 "Dating" (Valentine's Day Special; Due 2/17/2017)

Dating.png
  • Source: Paul Nahin (2008), "Digital Dice". Problem #20: "An Optimal Stopping Problem"
  • Problem: What is the optimal time point when one should stop dating more people and settle on a mate choice (and live with the decision)
  • Your best strategy is to date an initial sample of N individuals, rejecting all, and marry the next one ranked higher than any of your N individuals. The question is what is the optimal number for N.
  1. The problem could be investigated by simulating a pool of 10 individuals, ranked from 1-10 (most desirable being 1) and then take a sample of N
  2. You may only date one individual at a time
  3. You cannot go back to reach previously rejected candidates
  4. Simulate N from 0 to 9 (zero means marrying the first date, a sample size of zero)
  5. For each N, obtain the probability of finding the perfect mate (i.e., ranked 1st) by running simulation 1000 times
  6. Plot barplot of probability versus sample size N.
  7. Expected answer: N=4
Submitted Codes
  • By Weigang
pick.candidate <- function(min, array) {
  for (i in 1:length(array)) {
    if (array[i] < min) {
      return(array[i])
    } else {next}
  }
  return(0) # No 1. has been sampled and rejected
}
candidates <- 1:10;
output <- sapply(0:9, function(x) {
  ct <- 0;
  for(k in 1:1000) {
    if (x==0) { # no sample, marry the 1st guy 
      sampled <- sample(candidates, 1);
      if (sampled == 1) {ct <- ct+1}
    } else {
      sampled <- sample(candidates, x);
      not.sampled <- candidates[-sampled];
      not.sampled <- sample(not.sampled);
      if (pick.candidate(min =  min(sampled), array = not.sampled) == 1) {ct <- ct+1}
    }
  }
  return(ct);
})
barplot(output/1e3, names.arg = 0:9, xlab = "number of sampled dates", las=1, main = "Optimal stopping for dating (N=10 candidates)", ylab = "Prob(marrying No.1)")

(Previous week) Feb 3, 2017 "US Presidents" (Due 2/10/2017)

Sim-presidents.png
  • Download : 1st column is the order, 2nd column is the name, the 3rd column is the year of inauguration; tab-separated
  • Your job is to create an R, Perl, or Python script called “us-presidents”, which will
  1. Read the table
  2. Store the original/correct order
  3. Shuffle/permute the rows and record the new order
  4. Count the number of matching orders
  5. Repeat Steps 3-4 for a 1000 times
  6. Plot histogram or barplot (better) to show distribution of matching counts
  7. Hint: For R, use the sample() function. For Perl, use the rand() function.
Submitted Codes
  • By Mei
pres.list <- lapply(1:1000, function(x) pres[sample(nrow(pres)),])
cts <- sapply(pres.list, function(x) {
  ct.match <- 0;
  for (i in 1:45){
    if (pres$order[i] == x[i,1]){
      #cat(as.character(x[i,2]), x[i,1], "\n")  #as.character to avoid the factor info
      ct.match <- ct.match + 1;
      #cat(ct.match)
    }
  }
  ct.match #return
})
barplot(table(cts),xlab = "Number of matches per shuffle", border = "hotpink", col = "pink", ylab = paste("Frequency total of:",length(pres.list)), main = "US Presidents")
  • By John
import pandas as pd
import numpy as np
from pandas import DataFrame
import matplotlib.pyplot as plt

df = pd.read_table("presidents.txt", names=["num", "name", "presidency"])

name_list = list(df.name)

# create a list to store matches after each shuffle
shuffle_record = []

# create a function in which the first argument is the original dataset; second argument is number of shuffles
def shuffler2(original, n):
    record = []
    for i in range(n):
        num = 0
        each_shuffle = {}
        temp = original.reindex(np.random.permutation(original.index)) # do shuffling for each
        compare = original.num == temp.num
        matched_df = original.ix[compare[compare == True].index]
        for i in matched_df.index:
            each_shuffle[i] = matched_df.name[i]
        shuffle_record.append(each_shuffle)
        try:
            num = compare.value_counts()[1]
        except:
            pass
        record.append(num)
    return(record)
result2 = shuffler2(df, 1000)

plt.hist(result2, color="yellow")
plt.title("Histgram for President Data")
plt.show()
  • By Weigang
p <- read.table("Presidents.txt", sep="\t", header=F)
colnames(p) <- c("order", "name", "inaug.year")
# Use "sapply" or "lapply" for loops: no need to pre-define a vector to store results
p.sim <- sapply(1:10000, function (x) {
  length(which(sample(p$order) == p$order))
  }) 
barplot(table(p.sim)/1e4, las=1)
p.exp <- rpois(1e4, 1) # draw 10000 Poisson random deviates
mp <- barplot(table(p.exp)/1e4, las=1, xlab = "Num of matching presidents") # mid-point on x-axis
lines(mp[1:7], table(p.sim)/1e4, type="b", col=2) # add a line (Poisson-expected) to the barplot (simulated)
legend("topright", c("Simulated", "Poisson expectation"), col=1:2, lty=1)

(Future weeks) March 3, 2017 (Due March 10, 2017) Pi & E=