Today we are going to start using Pandas more. Today’s files are here https://github.com/bmaillou/BigDataPython/tree/master/10-StartPandas
We are using part of a data set that looks at arsenic concentrations in Bangladesh.
VIDEO
MAKE SURE TO TAKE THE QUIZ!
Homework
Due Next Class (Wednesday February 26, 2025).
Take the data from class today and plot depth on the y-axis and arsenic on the x-axis. I would use ax.scatter(). Because when you plot something versus depth you always want depth in reverse order or inverted so the 0 is at the top and the maximum is at the bottom. Use ax. and some google searching to find out how to do it. But if you stop and think about setting ranges you can change your start and stop intervals. Then the next step is to put the x-axis ticks on top and to put the label position on top. then you can set the x limits (xlim) to get the plot so it starts at 0. Also xlim could go from high to low. If something takes a list you add a [] inside the (). luckily to make this plot there is more than one way to alter all the parts. When searching for answers I would google something like “matplotlib switch axes direction”. I usually add pandas or matplotlib to the search and it should show you options.
3. Now the second graph is fancier. Take the first graph and color the points as shown below. Do this by calling scatter three times but only selecting the data you want to plot each time you call scatter. Think about the boolean calls we did in class. These are the arsenic concentrations I am coloring. Get the first graph nice and this second graph becomes easier. Do not copy my colors. These are good colors for representing the points. Choose three colors and three symbols of your own and see how they work. As a hint I would first do the less than 10 and then greater than 50. Then you can do the 10-50 which needs an “and” of some kind. Choose your own 3 colors and 3 symbols. You can use xkcd colors. You use them by saying ‘xlcd:color’ or just google color palettes. You can also use the hex designation. Then again you can just do the matplotlib colors.
Your are only allowed to add two lines of code to answer this question. Take your code from today where you read in Brian.csv and plotted the array. You can only add two more lines of code to your work from today to make the plot. Using two lines of code and your slicing nomenclature add an exclamation point. Remember in the file I had the background as 0’s and the writing as 1’s. You need to at least make a third color!. Then print it in a color of your choice using a colormap. My answer is below. Both answers are perfectly correct. I just wanted to add more color this year! To make the lines crisp I turned off interpolation. Here is the documentation. But in imshow you need to add the keyword argument interpolation=’none’ like we did in class.
2. For part 2. Make a new csv file in excel and make it so you can print your name in more than one color. That means Your name must be in multiple colors (so at least 3 colors overall). So read in your file and make the plot. By plot it I mean show your name like I show my name! As a hint. I made my csv file in excel and I made the rows and columns smaller or changed the zoom so I could see the pattern I was entering. When saving a csv in excel the ms-dos csv option seems to work the best. When making the csv file name the csv file with your name. Make sure to upload your csv file with your homework.
3. For the third part. Make an excel file with one column for x-values and three columns for y-values. Label each column. I would do ten or so rows. So approximately ten values for the x-values. For the y-columns make each distinct. You can choose your own function or pattern (straight, curved, etc) for the y-values. But make sure each line is unique. Now read in the file and plot the lines in excel and label the axes. So you need to make an excel file and use it to plot three distinct lines. Make sure to hand in this file.
For handing in make sure to hand in your notebook, your csv file for you name, and the excel file for your lines. If you ever reading a file in your notebook make sure to hand in that file. Make sure that file has your name in the title and works.
We have done some line fitting and we are going to do a lot more! Half the life of a scientist is taking data, plotting it, and looking for a correlation. Plus I want to make sure you can do this in your sleep. What we are adding now is we will start to read data from a file so we can handle much larger datasets. Today we are going to read in an excel file into Pandas which is the python library for line fitting and fit a line. We are going to look at GDP versus life expectancy. A common dataset in sustainable development. This is to the website where I got the the data but I also put the excel file on GitHub. Al\l files are here https://github.com/bmaillou/BigDataPython/tree/master/08-MoreLineFitting
1. You need to find some data from the web or a paper. It could be from a paper you have read. It could be from another class. Find data with at least 10 data points. I would do somewhere from 10-50 points. Do NOT use data with dates. Dates are hard. If you use time for an x-axis it should be hours or something similar but not months/days/years with slashes. We will learn how to do dates later in the semester. All other data should work.
2. Put the data in an excel file. Add your name to the name of the excel file. This way I will be able to differentiate and read all the files.
3. Read in the excel file and fit a best fit line. Remember to add a figure caption and all other parts of a figure. Add the text box showing m,b,r,p values to the graph. In the figure caption say if the result is significant.
4. Make a new sheet in your excel file that is a copy of your data. If you data had a significant correlation (p<0.05) change the data in the second sheet to make it not significant (p>0.05). Then read in the second sheet but remember you need the sheet_name keyword. Show the new correlation. If your data had no significant correlation (p>0.05) change your data to make it significant. Show this new changed plot with a best fit line, figure caption etc. I want you to manually alter your data in a second sheet and use the new data. Add the text box showing m,b,r,p values to the graph. In the figure caption say if the result is significant.
5. Make a third sheet in your excel file. Change your data again but this time take your significant correlation and multiple the y-values by -1. Make sure to take your significant correlation. Now save that sheet as a csv file with your name in the title again so I can read it in when grading. Read it in and show the best fit line, figure caption etc. Add the text box showing m,b,r,p values to the graph. In the figure caption say if the result is significant. Now answer this question. Hoes does the r-value, p-value, and r-squared value compare between the data and the data when you multiplied by -1. What does multiplying by -1 do? Be specific.
6. Summary
one figure of best fit line with your data
one figure of best fit line where you altered your data to make either significant or non-significant
one figure of your significant data altered by multiplying by -1 read in from a csv file.
7. For handing in.
Hand in your notebook
excel file
csv file
The excel file should have three sheets in it.
Make sure your name is in each of the file names.
If you are ever reading a file in your notebook make sure to hand in that file. Make sure that file has your name in the title and works.
DON’T MISS THIS NEXT PART!
8. This part will also be graded for the final 10%. As part of the class you have to comment on three of the lectures by using the comments section of the blog over the course of the semester. You do not have to comment now. You will comment as the semester proceeds. Comments can be wide ranging but need to be designed to improve the teaching or the homework given how you learn. Would you change the notebook? Would you change the homework. Is there a hint I should have given. How can you help me improve your learning and future students learning. Previous comments have already helped me to streamline the packets and homeworks. But now to make it fun you have to write an ipython script that randomly chooses your three classes. You need to randomly choose 2 classes between classes 2 and 16 and 1 class between classes 17 and 23. There are at least two ways to do this; you could use numpy.random.randint or you could use random.randint. my answer using numpy looks like
Brian will comment on classes [15 12]
Brian will comment on classes [18]
my answer using randint looks like
Brian will comment on class 15
Brian will comment on class 3
Brian will comment on class 21
Remember since you are using a random number generator everyone’s answers should come out different! If you get two of the same number re-run it to get 3 unique numbers! Plus every time you rerun the cell your numbers will change!
We are going to explore more how to do some math in iPython and how to make some plots. Today you are going to write your own program to get the best fit line. The equations are from Harris-‘Quantitative Chemical Analysis’ (Harris-fit-a-line). I am going to give you the equation and then using numpy you will make it work. Once it is working you will then go back and get your your Montauk and KLGA data and determine the correlation. Do not look at the answers until you are done today. Talk to your neighbors about turning equations into code. Getting the thinking and parenthesis correct takes time. Everything is here https://github.com/bmaillou/BigDataPython/tree/master/07-FitaLine
VIDEO
Make sure to take your quiz!
Homework 6
Homework 6 Due Monday February 17, 2025
You need to go and create a dataset and then fit a line to it.
You should present a hypothesis, get the data, determine the fit.
Your data and hypothesis can be anything and does not need to be “real”. you can be creative. You could interview friends to get data points. You can search online. You can make it as fun as you want. You can make something up! But you need to state a hypothesis. Make sure to clearly state your hypothesis. Have fun with it.
You need at least 5 data points that you generate from your ideas and hypotheses. Make it look good! This is not about the data. This is about the idea of presenting a hypothesis and fitting a line! make it fun! Be goofy!
Once you have the data make the graph look professional. Then you have to write a figure caption and summary sentences about your results in markdown. This should all be in one ipython notebook with nice Markdown.
For this I mean add a markdown cell below each graph explaining the result in a figure caption.
The results should be presented from both your method for fitting a line and a python method for fitting a line. This is redundant but make sure to do it! Fit the same data twice to make sure you get the same result. You should make sure that your line fitting algorithm and the python algorithm give the same answer.
For your method I mean where you did the math on your own.
For the python method I mean linregress.
The results need to be presented in the figure as a title or a text box. I like the text box! The text box makes it look professional.
Go back and find when you plotted the parabola and sin x. These are really nice graphs that you made.
Use input and if statements to ask the user if they would like to plot sin x or to plot the parabola
If they choose sin x use an input statement to determine the number of pi’s they want to plot and plot the appropriate graph. This way you can show many pi values.
If they choose a parabola have them enter the a,b,c of the parabola.
For example if I choose sin of x I could then choose 32.1 pi’s to plot.
For example if I choose parabola I could then choose a=1, b=-5, c=0.02
You should have all your code in one ipython box so I can just run that one box, do my entries, and then see the plots. if-statements grading rubric
Remember to use input()
To get an integer or float (remember to use them correctly, You need to choose when you think an integer or float is more appropriate) use input like
a=int(input(‘please enter a number as an integer’))
myfloat=float(input(‘please enter a number as a float’))
Download or take a screen shot of your certificate and then hand it in on courseworks under assignments. Your name should be on your certificate.
The second course is the 20 hour coding course https://studio.code.org/join/RCXQVG You do not have to do the unplugged sections. This is due 2/10/25. You CANNOT join both classes at once.
Joining these classes is quirky. Use the links above. You can also first sign in to code.org then go to https://studio.code.org/join then type in the two codes above. Check with me and I will make sure I can see your progress.
Watch Video for next class and take the Quiz.
Homework (DUE Monday January 29, 2025 ):
The Frozen Hour of Code.
Watch Next Video and Take Video Quiz!
Make Sure you Install Anaconda
Watch Growth Mindset on Home Page and take quiz.
Alternative plans
ON YOUR COMPUTER
On a Mac. UPDATE Just do the installer. If it fails we can do the command line.
This is critical. You need to do the command line installer. I am updating the video
Install like above. But to run I open terminal. You can open terminal by typing “terminal” in the search icon. Then at the prompt type “jupyter notebook”
On a PC
I run anaconda on my PC. It has a quirk. Install it. But then my launcher always crashes. To work around.
Instead of opening installer open the anaconda command prompt. If you can’t open this, go to the file browser and open a new “Anaconda Prompt” window.
once open type “cd /” to get to root directory. This gives you access to all your directories on your hard drive. I Can’t run from here.
type “dir” to get your list of directories.
change to the directory you want to use using cd and you are ready.
now type “jupyter notebook”.
You should have notebook now open in your browser
ON THE MILBANK 222 MACS
due to technical issues we can’t permanently install anaconda so we have to install it every class. the installer is on the root level of the hard drive.
Google Anaconda Python
Click on the MAC symbol and download 3.6
Click on the Installer and then go through installation.
Use 2 nested for loops. This is a for loop inside a for loop. Count up in the outer for loop from 0 to 9 and then at every step count back down to zero. Remember that in np.arange(start,stop,skip). You give it a start,stop, and skip. We usually put numbers in. But you don’t have to add numbers you could put a parameter in. Add spaces like below.
Go back and get the monthly precipitation from central park we used in class 3 (NOT class 1). Remember that we used this data when we were learning about lists and learning how to make plots. Use a for loop and sum up the monthly data. Then call the function sum on your data. It will be np.sum(precip_list). How do they compare?
Answer Question 2.
For loop sum= 45.13
sum function sum= 45.13
QUESTION 3.
Create two lists. One with the day of the week and one with the number of classes you have on that day. Use a for loop to go through your lists and printout how many classes you have each day. Then write your total number of classes.
On Sunday Brian has 0 Classes
On Monday Brian has 2 Classes
On Tuesday Brian has 0 Classes
On Wednesday Brian has 2 Classes
On Thursday Brian has 0 Classes
On Friday Brian has 0 Classes
On Saturday Brian has 0 Classes
In total Brian has 4 Classes
QUESTION 4.
Using np.arange or np.linspace and a for loop to recreate what is below. Remember that np.sin takes radians and not degrees. You can convert yourself or use the function np.deg2rad(). Also, I wrote out degrees and did not use the symbol because latex does not show up in print statements but it does on graphs. Don’t forget what you learned about format statements a few classes ago to get all the decimals correct.
The sine of 0 degrees is 0.00
The sine of 10 degrees is 0.17
The sine of 20 degrees is 0.34
The sine of 30 degrees is 0.50
The sine of 40 degrees is 0.64
The sine of 50 degrees is 0.77
The sine of 60 degrees is 0.87
The sine of 70 degrees is 0.94
The sine of 80 degrees is 0.98
The sine of 90 degrees is 1.00
The sine of 100 degrees is 0.98
The sine of 110 degrees is 0.94
The sine of 120 degrees is 0.87
The sine of 130 degrees is 0.77
The sine of 140 degrees is 0.64
The sine of 150 degrees is 0.50
The sine of 160 degrees is 0.34
The sine of 170 degrees is 0.17
The sine of 180 degrees is 0.00
Final 10%
Fun with two for loops, line breaks, new lines, and np.arange. Can you make the pattern below? Remember you can do math in np.arange.
Homework 3 Due Next Class (Wednesday February 5, 2025)
Code.org intro course through stage 15 to stay on pace
Write 50-100 letters of nonsense in a string. But in that nonsense have a secret code that spells your name with every x letter (e.g. every 5th letter).
print the whole string
use slicing to print your name
use slicing to print your name and then make it upper case
Repeat the same thing with a list but only do your first name. This is different as you are using a list. #2 above uses a string and this uses a list. It is a subtle difference but make sure to do it correct for each. Making a list is more annoying. So you only need to do your first name.
print the list.
Use slicing to print your name
Print your name in reverse using slicing (for this to easily work the last letter of the list needs to be the last letter of your name)
reverse the list
print your name backwards using slicing on the reversed list.
with the list reversed print your name forward with slicing.
Final 10% today. Practice some more plotting and make the graph at the bottom of the page. Your graph should be your first initial. Not a B. You should be able to put together your line plotting and cosine plotting to make it. The graph below shows cosine. I am okay with an ugly letter but it should be legible. You will need to create two and only two numpy arrays to make your letter if you set all the different coordinates correctly. Think back to your lists and line to think about how to make your letter.
My Answers
My secret coded string!
Boerrianion4an6nn09 i Jkaj.hif ksfMifnasgmi30flbn-lniao238uasdx
Use Markdown after the graph to add a figure caption. It can be brief but get in the habit.
This is my example but it is not perfect!!!!! something similar will get you a 50. You need to add the missing parts.
Final 10%: To get the final 10% of the HW credit can you make a second plot with a slope of 2 and an intercept of 2? Remember all the parts of the first graph. Also, remember copy and paste are your friends…… Also remember what slope and intercept are!
First we will practice changing directories and thinking about where we work.
Reminder. If you want to download it from github right click on raw and choose save link as. If you get a .txt file ending change it it .ipynb and you can now open it in your jupyter notebook.
Second we will do Markdown, basic math, and printing
VIDEO FOR TODAY
QUIZ for after Video
Remember that you need to take the quiz before class on courseworks! It is labeled quiz 1 and called
Quiz 1: Python Video on Directories and Printing
Notebook 1 Due beginning of Next class. (Wednesday January 29, 2025)
Code.org intro course through stage 5 to stay on track.
Look at the directory structure of your computer and make a place to work from when using Python (not graded, for your own organization)
In the Jupyter Notebook for your homework use Markdown to make a nice header with your name, date, assignment and other information that is relevant. This is required of all homework. The reason we use the notebooks is because you can add markdown to make programming understandable and reproducible. Start practicing that from the first homework. This is critical. You must add your name, date, section, and assignment in markdown.
Use one print statement to make a capital letter of your first name. My B out of b’s is below. In your print statement you can use your letter, \t, \n and minimal spaces. My example is below but you can make yours look nicer with your letter! You need to use one print statement only!.
Use print statements and .format to answer the math questions below exactly the same way with the exact formatting. You need to set an and b and use them in your format statement. Your print statement cannot have numbers. Repeat-There are no numbers in your print/format statement. You set the an and b before the print statement.
Upload your homework to courseworks. You upload the notebook. The file you upload should end in .ipynb. DO NOT upload a pdf and do not upload screen shots. There is no need to export anything. Just go find your .ipynb notebook and upload that to courseworks.
Watch Next Video and Take Quiz
Brian B (Really looks more like an 8)
Your answer should nicer than
BBBBB
B B
B B
BBBBB
B B
B B
BBBBB
Set
a=9.000
set b=4.000
Write the following sentence exactly. using the .format nomenclature and no numbers only variables. Use the a and b that you set in the format and print function. Use the different operators with your variables. Look at today’s handout for examples with the print and format functions. The * should be in with the formatting calls.
When 9 is divided by 4.00 the answer is **2*** with a remainder of 1.0
Then to check you did it correctly. I will open your notebook. I will change your a=55.000 to a=4.000 and it should give me this sentence
“When 55 is divided by 4.00 the answer is **13** with a remainder of 3.0”
