Classes 21, and 22 – NetCDF

Posted on by

This is our second to last assignment……..

We are going to expand on our maps and NetCDF to make an animated GIF.

These are all really helpful skills to have going forward.

Mapping is a critical part of earth sciences.  A lot of people will use google maps and ArcGIS for their mapping.  These are great tools. What i am also learning is that Python has a lot of tools within it that makes mapping very easy. Plus you can make professional looking maps.  Plus if you have multiple parameters in your pandas dataframe it is easy to loop and plot them all.  This is a new mapping segment as I changed what I used to do to update it and help us make publishable maps. I am still organizing so here is a list of files we will be using.

Files.

NetCDF is now becoming a common file type in the earth and atmospheric sciences.  After doing today’s workbook your homework is to make something similar to the animated gif below.  I put a ton of hints in the notebook and it really brings together everything you have learned.  Be patient.  This is a really big data set.  In addition, we are just using numpy arrays for this.  No need to use pandas!

Homework

Due See Courseworks

1..  Make the animated Gif below.   But you need your name on it somewhere so I know it is yours and not copied from someone else.  Also, make sure to write a brief description of what you observe in your Gif. Do you see any seasonal patterns?  Also, I am not sure why I lost resolution on my writing. I am working on it…  (Note to myself.  To get it to work I needed to make it full size).  I JUST LEARNED how to get better resolution on the words.  I did this.  fig.savefig(filename,dpi=50,bbox_inches=’tight’,facecolor=’white’).  When reading stackoverflow someone said putting a background on a png improves the letter quality and it really does help.

 

IGNORE BELOW THIS

Overview

This is our last assignment……..

We had to skip mapping but I will show you a little and this is a really fun project.  …. One of the things I wanted you to learn was how easy it was to install packages. We learned this with netCDF.

We need to install Basemap today.

conda install -c anaconda basemap

Then in our notebook we need to add these lines at the top maybe.

#On Mac's
import os
os.environ['PROJ_LIB'] = '/anaconda3/share/proj/share/proj'

First try without.  Then if you are on a mac. try it.  If you are on a pc and it doesn’t work we need to look at your screen.

NetCDF is now becoming a common file type in the earth and atmospheric sciences.  After doing today’s workbook your homework is to make something similar to the animated gif below.  I put a ton of hints in the notebook and it really brings together everything you have learned.  Be patient.  This is a really big data set.  In addition, we are just using numpy arrays for this.  No need to use pandas!

Homework

Due See Courseworks

  1.  Make a map of surface air temperatures on the day you were born and mark where you were born.

2..  Make the animated Gif below.   But you need your name on it somewhere so I know it is yours and not copied from someone else.  Also, make sure to write a brief description of what you observe in your Gif. Do you see any seasonal patterns?  Also, I am not sure why I lost resolution on my writing. I am working on it…

 

2. For the final 10% bonus….. You need to make a second animated gif.  I have two examples for you below.   Make ONE of them!  I think the sea ice on might be more straightforward.  Remember the data is from our notebook. To get to the data go here to noaa  and then scroll up and hit ‘select see list’ and then ‘see list’ again.

  • skip this one.  We can make the second animated Gif that compares 2014 and 2015 to see if we see the impact of El Nino.  Mine is below.  If you want you can choose two other years to compare.  It is not much harder………  But it takes some thinking and really slowed down my computer.  Give it a shot!  If you make it describe the difference you see.
  • Make an animated gif of the sea ice extent for one year with a polar projection.  From basemap you know how to make a polar projection.  Then it becomes very similar to what you have already done.  You just need to download the sea ice data from the same website.  The units are from 0-1 and are in fraction of sea ice.  So a 1 means 100% of the sea has sea ice.  I changed my colors this year in the example from class and I think I like it much better.

 

 

 

SKIP THIS ONE

elnino

42 thoughts on “Classes 21, and 22 – NetCDF”

  1. I had a lot of fun doing the mapping in this class, and I found that it did a good job of compiling many of the skills that we had learned over the course of the semester. It took me a long time to figure out how to get the gif to work but it was very satisfying once it eventually did. I agree with the comment below that mentions explaining the process of saving the images to an individual folder. Other than that, I think the class notebook was well organized and clear! As I used mapping for one of my figures for the final poster.

  2. I had a lot of fun doing this assigment. I think the only hard part of this assigment was to create the 365 images and add them to the same folder. Other than that, everything was pretty much straightforward.

  3. Unsurpisingly, this was my favorite class. I haven’t taken a GIS class before, so this was all very new to me! I didn’t even realize that map projects can be done on python. However, this assignment was super difficult to understand – the loop took me a while to figure out. I like that this assignment spans over a couple classes. I think explanations of the units for time would be super helpful in future classes. It took me a while to understand what I was actually doing when setting the dates.

  4. This was a fun class to end on! Making the GIFs was fun and it was also a good assignment because it used a lot of what we had learned over the semester. I liked learning about netcdf files as I had never used them before. Luckily, I never ran into issues with my code taking hours or my computer crashing. Overall, good class.

  5. i found this assignment really cool. I loved being able to see the GIFs I made at the end and know that I wrote the code that got them to work. I thought that the notebook was well done and easy to work through. I think more should be explained towards the end of the notebook when it comes to downloading all the images to a folder and compiling them to a GIF because I really struggled here and couldn’t figure it out for myself. I think this was a good final assignment to pull everything together and it made me see how much I had learned during this semester of coding and made me excited to try my final project.

  6. This was a very good assignment. The notebook was very well made so that I ended up not having too much trouble with making gifs. It was a little challenging at first, as it took a while for me to understand what the for loops were doing. However, it was a lot of fun to actually learn how gifs like these are made considering how often I see them in the world, and to be able to make one oneself.

  7. I loved that this was the final assignment because we see gifs all the time online, but rarely do we get to create one for science. I had a lot of fun working on the homework and the notebook — the only thing that was an issue (and I don’t know if there’s an easy way to fix this) was that each time I tried to run my code I had to wait almost an hour to see if it worked, which made it a very long and arduous process that crashed my laptop multiple times. Maybe in the future students could be advised to test their gifs by just creating the first five images? All in all, it was a great assignment and a good way to end the semester.

  8. I found this lesson useful because it allowed us to incorporate several concepts we learned throughout the semester, and because the datatime module and timedelta functions are are commonly in climate GIF maps. However, I think it would be helpful to include an additional hints/background on os.listdir because I was a little confused on its purpose in creating the GIF.

  9. I absolutely agree with prior commenters who said they got a big kick out of making the gif! So did I.

    From a “teaching data analysis” point of view, I would be interested in seeing more examples of netCDF files. I think the current description of netCDF files strikes a good balance between explaining some of what’s going on and not going into too much detail, but I’m further interested in the part about how netCDF “is used a lot in oceanography and atmospheric sciences.” I was really excited to learn about netCDF files because when I looked for oceanography data earlier in the class, I found it was all in netCDF files and I didn’t know how to work with it. Without examples like that people might not be so motivated to learn about netCDF!

  10. This was sooooo cool! I have taken a GIS class before, but I’ve never used code to generate maps or GIFs. Very useful skill for a visual time series analysis. As mentioned in the other comments, the start date of the NetCDF file is weird (1800-01-01), but once you figure out your way around that, it’s not too bad.

  11. This assignment was one of the funnest all semester! Like after making the gif I was just so impressed I had done that haha. It made me want to find some aspect in my senior thesis to which I could apply this new skill. However I would make the directions about using the gif generators a little more clear because on one of the websites I accidentally clicked the wrong thing (which was totally my fault) and got a virus on my computer which wasn’t great.

  12. This assignment was one of my favorite all semester. Like after making the gif I was just so impressed I had done that haha. It made me want to find some aspect in my senior thesis to which I could apply this new skill. However I would make the directions about using the gif generators a little more clear because on one of the websites I accidentally clicked the wrong thing (which was totally my fault) and got a virus on my computer which wasn’t great.

  13. I really enjoyed this notebook. I thought it was an interesting way to use for loops.
    However, saving each picture required a lot of memory and it would have been nice to learn how to make a gif in the python notebook itself (which was alluded to in the notebook), even just as an extra bit at the end.

  14. I really liked this class a lot! I think this is a really useful way to learn to display information. I felt like this class was a good culmination of many skills we learned–maps, loops, saving files, ect. The only issue I have is that I still don’t feel like I totally understand timedate stuff. I’m still unsure what we were actually doing with it, I more just followed along the instructions and saw the results but did not get the theory.

  16. This was among my favorite classes, I will suggest that we talk more about how much space netcdf files can take up on a computer. My python notebook crashed a couple of times because of the amount of iterations needed for this assignment. I also think you should check to see which GIF makers are proven to give viruses to computers. Otherwise I really enjoyed this class and it was a great way for us to use matplotlib, maps, loop and a bunch of other techniques we learned in class. This was the ultimate final project, both difficult in its comprehensive assessment of what we’ve learned this past semester but also really fun because of the graphics.

  17. Like the mapping assignment, I enjoyed this GIF assignment because of the immediate visual feedback according to each thing I changed. However, it was a bit frustrating that the hundreds of files were first arranged alphabetically and we had to change the file names in order for them to be ordered by day and month. It would have been nice if we had been warned about that before we created all the files under the wrong name, only to face frustration that we had to create them all over again. Especially because the high volume of files made my computer slow and unhappy 🙁

  19. I also really enjoyed this project! It was very cool to finally see where all the mapping, gif-making, png-saving were going. As a visual person, I thought this was the most attractive homework to me–particularly challenging at moments, reading netCDF data into a visual was powerful, appropriate, and made sense.

  20. I thoroughly enjoyed this class. I felt it was super applicable to current social media to understand how GIFs work. It was interesting and also a good review to combine what we learned about for loops, date time index and basemap all together in one lesson. Then on top of that, we were able to make a really cool image file out of that.

  21. I really like this assignment. Although i though this was the hardest one. It incorporated a lot of things we learned in class with was really good. One thing that i think would have been helpful was to have something like Q-A in piazza. I found that very helpful for other classes. I think for this class it would have been very useful to post questions on the codes we were using and discussing with other people in the class.

  22. This class and homework was possibly one of my favorites as we used a lot our skills that we acquired throughout this course. It was great to see the wide variety of things that we can do through python on top of using makeagif.com. This was definitely challenging because not only was the code difficult to figure out but then it required saving the images in a correct format and reorganizing them in a folder prior to uploading them to make the gif

  23. Wow! This was a fun project — the piece de resistance of the course. Time intensive and with some unclear instructions (it’s not obvious that one needs to create 365 GIFs), it nevertheless taught how to create an animated visualization of global sea surface temp data. Very powerful technique.

  24. I thought this was a great lesson, and really cool to put to practice something so neat (like making a gif). I will say that I enjoyed this lesson being drawn out over multiple classes if for no other reason than that downloading the data and running the script where it prints all the pictures are VERY time consuming. It might be helpful if you advise students to download the data before coming to class (or even have them start the download the previous class) so that we can get straight to the grind. Also, the end of the GitHub booklet was not very helpful. It gives very little information (or much less even suggest) on using a for loop to save all the different files. Maybe that was on purpose to test our skillz tho….

    Elias

  26. I think that making a GIF was a really, really great homework assignment. Although this was the homework assignment that I spent by far the most amount of time on trying to figure out and trouble shoot, it was fun to talk about once I had completed it, since GIFs are something that we see all over the internet! This is another instance of when you showed the relevance of what we were learning in class by having us create something we would want to create outside of class.

    As a small note, maybe you should write “but you just need to type conda install netcdf4 h5py at the prompt *in terminal.” Because I didn’t know anything about installing packages and tried to type that install note into my python notebook.

  28. I liked these classes a lot – super rewarding to be able to brew from scratch something so useful and cool looking.

    I thought it was pretty awesome how we could get our hands dirty with a pretty complicated/academic filetype by learning just enough to be able to create informative maps.

    I experienced some issues when making the gifs online (like really small output size), though I did settle on a good site eventually. I found that the PIL package we used developed a script to generate gifs from images, that might be helpful for future classes:

    https://github.com/wnyc/PIL/blob/master/Scripts/gifmaker.py

  29. This was my second favorite assignment (mapping was my first favorite)! I loved creating the heat map gif because you could really see your work “come to life” in a way a static graph or map does not. To be honest, the hardest part of the assignment was using the gif website because whenever I uploaded the 365 images onto the gif website, it would either get stuck while loading or print out the pictures backwards. I tried nearly everything (changes the name of the png files, one-by-one dragging the pictures into a new order), but nothing worked… In the end, my gif just ran backwards but it still looked fantastic! I also had a little trouble formatting the date at first but then I asked my neighbor for help… All in all, a really exciting assignment with a fun, visual outcome!

  30. This was a superfun programming class! I especially liked how you introduced us to a new filetype. I would have liked for us to learn how to read in a list of files using a function, compared to a for loop. The read in methods we were introduced to in the pandas-classes were very helpful, but one more method would have added even more our programming toolkit.

  31. This was my favorite hw assignment! It was cool making something that I use all the time. One thing that confused me was how the netCDF file coded time. Why is January 1, 2014 coded as 78162? Why can’t it be 01/01/2014? Why is 01/01/1800 the start date, if the earliest data is from 9/1/1981?

  32. This exercise was helpful because it is important to understand what netCDF files are and how to work with them, especially because of their prevalence in the sphere of environmental data. Also, it was a really fun way to visualize data, and it helped with my understanding of plotting images in python, which was an important review before we were assigned our projects. And of course, it was fun and interesting to learn how to make a gif.

  33. Datetime module makes it easier to loop through the SST data to create maps especially when the netCDF4 file has the date saved in an unusual way (e.g. having time value of 78162 for January 1, 1800).
    It takes so long to create 365 maps and put them together in a gif. So if would be good to emphasize in the notebook (as you did via email) that we should first test the code by looping through smaller sets dates.

  34. I had some trouble figuring out this assignment. I think I got tripped up setting up the for loop, because I wasn’t sure how to incorporate the “enumerate” function in the hint, and I ended up not using it at all.

    In addition, I was also confused about saving the image if it was not a PDF.

    Third, I had trouble making the GIF. The website that we used crashed several times and I tried a few times. Other websites wouldn’t take files that large, but then I finally got it to work on a different website. It was useful to save the files so that the month wouldn’t be alphabetical but rather numerical, which totally avoided any possibility of the images being arranged in a different order than they should be.

    (I also posted my comment for class 7)

  35. It would be helpful to point out that when making the actual GIF online, to increase the speed of the GIF. I wasn’t aware at the speed the preset GIF is, and mine is consequently very slow. Since it took so long to create though, I didn’t make a faster one.

    Happy Thanksgiving Break!

  36. This was one of my favorite classes. It was such a simple and elegant way to visualize a ton of data in a really approachable way!

    I actually liked how for this class you gave us the basics, but didn’t necessarily walk us through each step. It was really rewarding to be able to get through the assignment independently (with some ups and downs of course, but working through each time I got stuck). After this assignment especially, I realized how much progress I’ve made in this class and how much I’ve learned! This project brought together a ton of basics (making nice looking plots, labels, color maps, for loops of course), and added the new data file type of NetCDFs (and related packages).

    And I know it was for SST maps, but making a gif from scratch felt snazzy since gifs are all over the place today (usually not as informative and scientific as these though!).

  37. Hi! I’m getting this error when trying to format the date time:
    “ValueError: year=1800 is before 1900; the datetime strftime() methods require year >= 1900”

    Does anyone know a workaround to format the year so it looks nice on the image?
    Thanks!

  38. It is good to know about NetCDF files. I found this lecture particularly useful for creating quick maps that don’t require basemap and cartopy. How do you create a NetCDF file though? And what is the benefit of using this type of file as opposed to a standard Excel file? In this lecture, we were also introduced to the datetime module and the timedelta function within datetime; datetime will certainly be most useful for future date and time based data. This next question is more about how the dataset was set up, but why use 01/01/1800 as the start date? Using 78162, 78163, etc. as a reference for time seems very inaccessible to a user unaware of the datetime module.

    • I also find the date units weird. But that is how people do it. Excel doesn’t similar in the background to do its dates also. But also, the netcdf files are in binary to make the file size smaller. this file is way too big for excel and could never be opened in it. People say once you get used to using the file format it is not too bad. Especially if you do a lot of work in that area.

  39. I am having trouble figuring out how to use datetime in the for loop.
    I was going to use “for day in range(0,365)” and then see if I could plot sst[day+78162], but that does not seem to be working.

    I am also confused about how to save each image as a png. We have only done pdf’s so far.

Leave a Reply