Blogs

After all that awesome time I had during SAGE DAYS 25, I still feel there is something I haven't done. There were many people asking about how to use SAGE. Although they have seen how to use notebook, they are not sure how to run a sage notebook server. This blog post is intended to enable a layman start with using sage.

This blog post will be in three parts

• How to get sage
• How to use sage on the terminal
• How to start and use a notebook server

How to get sage:

Visit this link and get the corresponding binary. Follow the instructions for extracting the binary. I now assume your sage is in a directory called SAGE_DIR.
Ex: my SAGE_DIR is /home/nishanth/sage-4.5/
From now on, where ever I use SAGE_DIR please replace it with your sage directory.

How to use sage on the terminal:

Using sage interactively:
First go to your terminal and do "cd SAGE_DIR"
then do "./sage"
it takes a while and gives you sage prompt. Which means you will see "sage:" on your terminal waiting for input.
This is the sage shell and you can type your sage code interactively line by line over here.

Running a file that contains sage code:
Simply pass the path to that file as an argument. In simple words, also type the path to file after you do "./sage"
Ex: ./sage /home/myscripts/mycode.sage

If the filename ends in ".sage" the file is run as a sage code. If the filename ends in ".py" the code is run as a pure python code.

How to start and use a notebook server:

Use notebook at home:
type "notebook()" on the sage prompt and it starts the server. Go to your web browser and open the url "http://localhost:8000" and go ahead.

Use notebook as a server for academic purposes:
A single instance of server can handle only around 30 - 40 simultaneous connections. So you might want to run multiple instances and just notebook() will not suffice.

Use notebook(interface="", port=<your desired port>, directory=path_where_to_store, timeout=120)

Ex: notebook(interface="", port=8080, directory="/home/nishanth/sagenb_8080", timeout=60)

This server can be accessed by "http://<server ip>:8080"

To run next instance, use a different port and different directory.

Date - 1st April, 2010
Venue - A1-A2, Mathematics Dept., & KRESIT, IIT-B
The session started at 10:40 AM with a brief welcome to all the participants from Prof. Kannan Moudgalya. Then each of the participants gave a brief introduction of themselves to everybody assembled. The participants mainly consisted of B.Ed teachers and student-teachers from Mumbai and near-by areas. A few of us from the FOSSEE team, Vivek Khurana, and Ganesh from SPACE were the only programmers, who were participating. Dr. J. Shah and Dr. Manojendu, domain experts for Netlogo and Geogebra, respectively, were present to help and guide the participants.
This was followed by a short talk by Prof. Kannan highlighting the importance of Spoken Tutorials for a country like India. He started with a short demonstration on how simple it is to make a spoken tutorial. Using this, he highlighted, what he felt were the advantages of a spoken tutorial. Firstly, a well made spoken tutorial captures everything and doesn't leave anything to the imagination of a viewer, and eliminates problems of mis-communication. Secondly, it can be
prepared using relatively easily available equipment - A PC, and a mic. Thirdly, anybody can do it - from kids to housewives to retired people. Hence, it is a highly scalable thing. The talk lasted for about 20 mins and put the workshop in perspective and was a nice way to start.
Next was a talk by Nancy, giving the participants a guide on how to write a script for a spoken tutorial. Most of the things were common sense, but it's nice to have them stated explicitly. Notes [slightly modified] that I took during the workshop are at the end. This presentation also lasted for about 20mins.
This was followed by a talk by Dr. J. Shah on Netlogo. He showed a small demonstration of Netlogo and followed it up with a list of ideas he had for spoken tutorials. He also shared a few scripts that he had started working on, but, not completed. Next, Dr. Manojendu gave an introduction to Geogebra and a few ideas for spoken tutorials. It was already noon, by the time we reached the end of the talks and the hands-on part began.
The participants were then divided into groups of three and they chose one topic each, from the pool of ideas to make spoken-tutorials on. We then began working on our scripts. Our group chose, "Introduction to Arrays" in numpy as the topic. I already had some outlines ready, which we worked on improving. It was good to have a novice in our team, who could review the script. But, this tutorial being a part of a series of tutorials was a slight problem for us. We had about 45mins
before we dispersed for lunch.
Post lunch, we had another half-an-hour to forty-five minutes to review finalize our script, and try out some rehearsals of recording. Then began the series of live recordings. Each of the teams were given an opportunity to record their scripts in-front of all the participants. It was an exciting thing for the participants and they were enthusiastic. But, a few of us felt, this was a bit too long and more time could've been spent in the script finalization and rehearsal stage with more critical feedback being given to the participants.
The workshop ended with a few participants giving their feedback. On the whole it was an educative workshop, but I would've liked more critical feedback, with fewer specimens.

• Steps for script writing
  1. Identify the need
  2. Prepare
     1. familiarize self
     2. test all the features
  3. Focus on a key feature that you want to demonstrate
  4. Choice of words
     1. Use short sentences
     2. Use simple words - avoid jargon
     3. Use a combination of screen-cast and slides
     4. Be specific
  5. Follow a pedagogical approach
     1. Brief intro
     2. OS compatibility, system requirements
     3. Show screen-shots of download sites?
     4. Mention pre-requisite knowledge
        Ideally that material should be available as another spoken tutorial
     5. Write ALL steps
     6. REMEMBER - Visual medium - SHOW
  6. Review the script for accuracy and clarity
     1. Ask a novice
     2. Improve until novice can follow until novice can follow
        without help
  7. Review the script for grammar
  8. Time duration
     10 minutes
  9. File size
     ~ 10MB
• Narration - Dos and Don'ts
  1. Speaking
     • politely
     • do not give orders
     • make it a pleasant experience
     • speak naturally
     • speak clearly and slowly
  2. Tips to cut external noise
     • closed room with min. disturbance
     • switching off the fan helps
     • close windows
  3. Dummy rec. for 2-3 mins to finalize rec. parameters
     • predict file-size
     • check clarity of audio narration and video
     • full-length dummy narration will help improve tone of
       voice and pronunciation
  4. Final narration
     • do it in one go
       • keeps tone of voice uniform
       • reduce editing time
  5. While dubbing
     • if possible, remove original audio stream
     • else, just add another stream (not more than 2 streams)
     • audio-video sync
     • meaning should be same
     • file size remains (almost) same
     • recording dimensions

Here is a link to a post by Nancy on the event.

When we first opened the registrations to the students, we didn't expect such an overwhelming response. We had close to 90 registrations in 2 days. By the time we reached Jaipur the number had increased to a whopping 143! It seemed like students had registered over night too. We were a bit concerned about the number since the lab could accommodate a little over 70 people. Thankfully the number of people who actually turned up were close to 75. We had a handful of post graduate cum faculty participants as well coming in for the sessions.

We saw a consistent drop in the number of participants as the sessions progressed. For the post break session the number had come down to 66. Post lunch sessions saw the numbers further come down to 57. But those who remained showed good interest in whatever was being taught. The second day turn up was 42 and almost all of them remained till the end. The response on the second day was better than the first since all of them present were genuinely interested.

The lab structure there was the best we have seen so far. It was ideal in the sense that it had 60 systems all in good condition, all facing on the same side. The lab was spacious and well ventilated, although it required more cooling(thanks to the Jaipur summer). We felt the need for a mic since our voices weren't reaching the last few rows.

One interesting that we observed here was that the participants were quite comfortable with typing and making fewer mistakes. As a result we saw that few people had written close to 300 lines of code in a day without having closed the IPython shell even once.

The quiz performance here was the best among all the workshops we have conducted till date. There were a few of them who were quick enough to try out some of the quiz questions on the IPython shell immediately as well :-P.For the first time we saw people scoring full marks on the quizzes. One guy, Rishi Mehta, requires a special mention since he scored full on all the three quizzes we conducted.

We did skip a few sessions due to lack of time and little interest since they had little relevance to Computer Science students. But in the end the workshop turned out to be a very satisfactory experience. The feedback seems to be very positive. We hope the students continue to show interest in Python and contact us and stay in touch by telling us how they have been working on Python.

Pics from the workshop:

The SEES Book Application is a simple application similar to django book and hg book. It has the features that enables a user to comment on all the paragraphs of the book.
Although the application uses reST format for editing, it converts it into a docbook intermediate. This docbook intermediate can be used for publishing the book into pdf format. Thus this app makes a great idea for web publishing and thus provides a discussion framework for various parties involved in creation of a book.

At the heart of it is a web application written in django that keeps a database of all the comments made on different paragraphs. Since the database of the number of paragraphs can change over the course of writing the book, it is kept in the form of python dictionary which is created every time the book is rebuilt. The commenting system is handled using javascript similar to django book.

Since we need to take care of the dynamic nature of the book during the discussion period, there is a build system for the book. The build system ensures that comments are not lost over the course of the discussion. It uses common tools like xsltproc and rst2xml/docbook converter in python docutils. xsltproc is used to convert xml/docbook to html using some xsl stylesheet. There is a tremendous scope for improvement in the app. It requires code refactoring, beautifying and making the code more readable and configurable . Generalization and making it more applicable to other similar use cases. First of all the docbook converter is specific for this project. A generalized docbook converter in docutils will be great . Their is work being done on in the docutils sandbox. Also a workflow design that automates indexing of chapters is required. A more generalized app can become a framework for book publication related discussions.

The url to the application is http://fossee.in/review/ and the code for the application is available at http://fossee.in/hg/SEES-hacks/

Videos of keynote by Dr. Travis Oliphant and interviews of various invited guests are now available here. We are yet to get videos of various talks of other participants(processing of videos has delayed things).
There are some problems with videos also, the keynote address and introduction seems to be playing sound only with right speaker. And then in interview videos, the introduction soundtrack for interviews is really sparky. /me says, Alll izzz Well

This year we got one workshop on SciPy-Image processing selected for Gnunify. We were waiting for schedule eagerly and when it came out we were totally confused. There were almost 6 parallel tracks. So we reached to Pune at night and after dropping the luggage headed straight to event place to make sure we can get Lab setup properly. There were lots of volunteers running around fixing up things, making sure that "all izz well" for next day.
We had talk scheduled on first day itself that too at 10 AM. All the rest of talks and workshops were staring at 10:30, we were the odd one out, and even that didn't help. It was Friday, a working day so I think many students and professionals might have skipped first day. By 9:50 everything was very relaxed, no hurry no rush, people were turning in slowly and figuring out what sessions they would be attending. But for us it was going to be tough to lure attendees. By 10:10 we had two participants turned up and to make things worse, we had Kushal Das covering basic Python right next door. That talk was to start at 10:30 and students were lined up from half hour before itself. We were thinking of going around advertising the workshop to get more people, or changing workshop to a informal session of hands on, but then suddenly out of no where we got precious interested attendees, decent 20 odd turn up. But still we made the anchor(they introduces speakers to audience) also to participate.
Now the workshop requirement was that participants should have basics understanding of Python, but we got students still griped in evil clutches of TurboC++(Bhagwaan sab ko sadbuddhi de). So we changed the pace and plan of workshop, we covered basics image reading, displaying, numpy arrays, slicing and striding concepts, and then processing image by using filters, dropping information and other fundas. Thankfully we did not notice any one dozing off, they were responding to small exercises, and doing hands on. All in all it went well.
But I think, Gnunify management could have scheduled talks considering kind of audience registrations they had. I think, if this workshop would had been scheduled after introductory workshops and talks we could have got more and better literate participants, hence we could had covered more. Okay would have, could had are always there, lets hope we next time it is better.

Vim Forever...
It all started when I saw Venkatesh Choppella working without taking his fingers off the keyboard to do many things which usually required shifting my hand between keyboard and my tailed friend, mouse. I knew for sure that he was using emacs. It was cool and I was fascinated about this cute little tool. My friend Puneeth uses it all the time for editing. He also has some crazy extension on firefox that troubles me while using. It is emacs extension.
Anyways now I wanted to use something cool like that. I know for sure that Vi is the competitor and so I started by googling for editor wars. I found many links that support emacs. So I started learning emacs rather looking at emacs. First thing I didnt like was the use of control key so often which required shifting of my fingers way down the home row. Then I had problem shifting my fingers to use the arrow keys for navigating through the file.
So here I am left with no other option than vi. I have used vi for quite a while and am pretty comfortable with it. But all I do in vi is first get into insert mode and edit the file as if I am using notepad. I started searching for tutorials on vi and found one that is so simple yet so amazing. I thank Swaroop CH, yes the same guy who wrote the book "A Byte of Python", for another amazing book "A Byte of Vim". It was so amazing and I started using vi like a novice vimmer.
Now editing a text file is something I do only once in a while. Most of the time I either code in Python or browse on Firefox. Ofcourse watching stuff and listening to music is also there but doesn't make sense in this context. So now what is this emacs plugin for firefox and is there something similar for Vim???. Yes there is one and it is called Vimperator. It transforms firefox into something that behaves similar to Vim editor and hence very seldom do I need the mouse. It is a totally different experience using firefox in Vim mode.
Now comes the most important job. Making Vim an ide for Python.
Let me start by listing a few things one does on a typical ide.

1. A file browser that stays on left of screen and enables opening files on one click
2. Ability to show multiple files in one screen
3. Syntax highlighting and code folding
4. Taglist that displays all the functions, classes and stuff in one place
5. Auto Indentation
6. Code completion
7. Debugging

Phew!! finally the list ended.Lets start with the first, The File browserVim has a very simple and light file browser plugin called NERDTree.Installing is as simple as extracting the files into ~/.vim directory.Using :NERDTree brings up the nerdtree and a tree structure is displayed.Moving between files is the same as it is in normal mode. The usual hjkl work and <CR> or the enter key is used to expand or collapse a directory. Clicking enter on a file opens the file in edit mode. 'o' key can also be used to do the same. <ctrl+w> <ctrl+w> is used to switch context between nerdtree and the file buffer. 'i' opens the file in a new screen.
Showing multiple files.This is as simple as splitting the screen into parts and showing different files on each screen.:sp splits the screen horizontally and :vsp splits vertically.If you notice, we have seen something similar while using nerdtree. What it is doing is simply opening another buffer for itself.<ctrl+w> <ctrl+w> can be used to switch context. The usual vim way of prepending a number also works here. so 4 <ctrl+w> would execute <ctrl+w> command four times. can also be mixed with hjkl.
syntax highlighting and code foldingvim takes care of syntax and all we have to do is :syntax on. You can also download a syntax file from here or write one of your own and name it python.vim. Installing is as simple as placing it in ~/.vim/syntax folder.code folding is a little different for different contexts. For python, the obvious is indentation. First do :foldmethod=indent. Then scroll to your block of code and za toggles the folding of code.
taglist is a plugin for vim that supports many languages. It requires a package called "exuberant ctags" which should be directly available on repo for any distro. Then download the zip file and extract contents to ~/.vim and you have your plugin.The visibility of taglist can be toggled using :TlistToggle and it works similar to nerdtree.
Autoindentationfollowing pep8, here are a few settings that should make your life easy.

• set expandtab
• set textwidth=79
• set tabstop=8
• set softtabstop=4
• set shiftwidth=4
• set autoindent

at this point you will notice that you have to type the 4 spaces or tab for the first time and then the code is auto indented from the next line.Wouldn't it be great if we didn't have to bother about even touching the tab key and the editor does that depending on the context. All that is just one step away. Download this script and place it in ~/.vim/indent folder. Now use the command ":filetype plugin indent on" or add the line "filetype plugin indent on" to your .vimrc file in home folder and there you go. Vim does all your indenting.
If you would have noticed, we have to type in all these conf settings every time we start vim. We don't want vi to do these settings for all the files either. What if we could set things only for files that end in .py. There is a very simple way of accomplishing that. Change the "set" to "setlocal" and put all your settings in a python.vim file and place it in ~/.vim/ftplugin.So your python.vim would look like this:

setlocal expandtab
setlocal textwidth=79
setlocal tabstop=8
setlocal softtabstop=4
setlocal shiftwidth=4
setlocal autoindent

Your ~/.vimrc file would like

" a comment
" display line numbers
set nu
" do indenting and stuff based on filetype
filetype plugin on


Phew, yesterday, rather today was a crazy night. I was trying to put up a django application for registrations of various users. Start was good and quickly within two days I got a decent application with captcha and other required fields. I uploaded it, and without much pain I got registration page up yAy.
We always have a local Point Of Contact from college where we are organizing the workshop. Now it will be nice feature if we can allow him/her also to view the registrations. And even better would be, he/she is limited to all the registrations they are in charge of, not of all the registrations. So again django views are simply great and I wrote simple view to limit user to particular list only. Nishanth is also learning django for his PyTasks so I thought it would be good idea to work together for some while to enable this view. He will get idea of how things work and I can get help from his learnings. He came up with this awesome one-liner to read a file, line by line, split it on particular delimiter and then create dictionary out of two fields

dictionary = dict([line.strip().split('|') for line in open('filename')])

Okay so everything working as expected, we tried to run the code locally and it was smooth and impressive(heh some time self-appreciation is good :P).
Now we pushed the code to the server, made sure everything was okay and in place, reloaded the apache and refreshed the page and BOOM gone. The normal registration itself vanished and we were getting error message of django not able to load views.py function from the app. It was bad, and people are still using it for registrations. It was already 1 am, we thought it would be over by 1:05, we will have some snacks and a good night sleep. But this BOOM really scared me. So we sshed into server, tried using django shell, loaded views and it was working fine. Nishanth tried extensive googling to search for some solution and came across a thread which mentioned about some problem with django, wsgi and reloading of views when something is changed. He said lets restart apache which I ignored for a while. We kept on trying making some changes and hoping that this time it will work. We were about to revert back to previous working version of site repo(Yeah hg always helps in such grave situations) when Nishanth said, "lets try restarting apache". We did that and yAy we got the app up again and running. Although it was not fault of django directly, but still I will give it also some credits. Phew by that time it was already 2:30 in night and we were like, dude we should hurry up, H-8 canteen is about to close and we wont get anything to eat if we are late.

The first version of Indian Scipy Conference was inaugurated on 12th December by Dr. Prabhu Ramachandran , Dr. Travis Oliphant (President, Enthought) , Mr. R Narayan (Vice President TCS Learning and Development), Prof. Elizabeth Sherly and Mr. Jarrod Millman in Technopark, Trivandrum . The Conference was organized by FOSSEE, Space-Kerala and SIG-FOSS. The aim of the conference was to create awareness and spread the use of Python in the Indian Scientific community . Also it tried to get the developers from India and abroad to tell about how they were using Scipy and Python for their research needs . They also brainstormed and sprinted on improving scipy documentation and code .

Mr. R. Narayan, former Vice-President TCS(Learning and development) started the conference by talking about object-oriented programming language paradigm and how high level languages like Python have made programming more accessible to people from different backgrounds. Then Prof. Elizabeth Sherly talked about importance of FOSS in Education.

Dr. Travis Oliphant of Enthought and one of the chief architects of Numpy kicked off the talks about how he started using Python for his research needs in the year 1998 . He traced the development of Numpy starting from Jim Hugunin's work on numeric . He also talked how open source development works and multiple people from different geographical locations collaborated to create Numpy by contributing small projects they were developing. He also showed that how Scipy and his company Enthought was working in diverse fields like embedded systems to Oil industry and food industry.

Prof. Kannan Moudgalya talked about his work with spoken tutorials at IIT Bombay. He shared his vision of freeing computer education through FOSS tools and spoken tutorials delivered in all the regional languages. Spoken tutorials could be used to bridge the digital divide. After that Chandrashekhar Kaushik talked about his Python based SPH framework and how it can be used to do simulations of particle interactions.

The second days talks were started by Christopher Burns from University of California, Berkley. He talked about NiPY, an open source neuro imaging tool developed in Python. NiPY project is an environment for the analysis of structural and functional neuro imaging data. He believed that Scipy and Python can function as great tools for scientific computing and hoped that more scientists and engineers will start using Python . David Cournapeau, the maintainer of Numpy project talked about the progress made in building numpy on different distributions and the feature set that will be added in the next few versions . He also introduced people to toydist a build system he was writing which tried to rectify the problems in building code using setuptools and distutils .The Chaco talk by Dr. Travis Oliphant on the second day got a lot of interest among people because of its ability to create very useful and interactive plots. He showed the power of Chaco by plotting the frequency of his voice using Chaco. Farhat Habib of IISER talked about using Python in Data Mining and Data Visualization especially in relation to gene regulation. Dr. Prabhu Ramachandran introduced people to Mayavi, a 3D visualization tool written by him. He also introduced Sage , an open source mathematical software. He showed Sage notebook's features like solving symbolic expressions and creating plots within the notebook. He also told people about the upcoming Sage days 25, India in August. In the end he thanked all the speakers, organizers and participants for making Scipy India successful.