Scientists and engineers: python advantages

Thread Starter

someonesdad

Joined Jul 7, 2009
1,583
A week or two ago a discussion in another thread made me think about writing a document as to why the programming language python and its add-ons are good tools for a technical person to have in his toolkit. I'm not suggesting it should take the place of other languages (indeed, an experienced technical person will know many languages), but I've found it an excellent general-purpose programming tool and it is usually what I turn to first.

I wrote the attached document to introduce some of the features of the language and discuss some of the add-ons that provide extra power. You can also get the zip file here (see the bottom of the page).

Edit 17 Nov 2010: I updated the document to add another example and discuss the problem of choosing between python 2.x and 3.x in more detail. The latest version will always be here: http://code.google.com/p/hobbyutil/downloads/list. I recommend getting the file from this link rather than using the file attached to this post.
 

Attachments

Last edited:

rapidcoder

Joined Jan 16, 2011
37
1. Scala is statically typed. Python has no static types. Which means if you make a type error, it will fail at runtime. If you make an error in a rarely taken branch, you may not notice the error when testing. In Scala such errors are detected on the program startup.
2. Scala is about 20-100x faster than Python to execute. Python is interpreted, Scala is compiled to machine code.
3. Scala has pattern matching, Python has nothing comparable.
4. Scala gives you more freedom in how you lay out your code, there is no meaningful whitespace.
5. Scala IDEs have code completion and refactoring support. This is not possible for Python.
6. Scala code is usually just as short and concise as Python (or sometimes shorter), albeit faster to type in because of auto code completion.
7. Scala has more mature libraries (any Java libraries work with it).

Enough?

Ok, Python has one advantage over Scala:
1. It is better for scripting short tasks. Scala scripts just take too much time to start.
 

tom66

Joined May 9, 2009
2,595
1. Scala is statically typed. Python has no static types. Which means if you make a type error, it will fail at runtime. If you make an error in a rarely taken branch, you may not notice the error when testing. In Scala such errors are detected on the program startup.
Dynamic typing brings advantages, and disadvantages. For example, you can create methods which can handle any type of members. Say an addition function; it can handle ints, floats, and any other type of item which can be added to another (as long as they are compatible - for example, adding an int and a list doesn't make sense, and will rightly raise an exception.)

2. Scala is about 20-100x faster than Python to execute. Python is interpreted, Scala is compiled to machine code.
Heard of Psyco? Python code can run in a JIT, which is almost as fast as native C! For most applications speed isn't crucial. If you need speed, you shouldn't be writing in either.

3. Scala has pattern matching, Python has nothing comparable.
See module "re" for regex, "glob" and "fnmatch" for basic wildcard (Unix-style) matching.

4. Scala gives you more freedom in how you lay out your code, there is no meaningful whitespace.
Python will let you write in any whitespace style you like, and frankly, you should be using whitespace anyway.

5. Scala IDEs have code completion and refactoring support. This is not possible for Python.
List of Python IDEs.

6. Scala code is usually just as short and concise as Python (or sometimes shorter), albeit faster to type in because of auto code completion.
See above.

7. Scala has more mature libraries (any Java libraries work with it).
JPython or Jython lets you run Java libraries from Python and the Python libraries are pretty good in my opinion anyway.
 

rapidcoder

Joined Jan 16, 2011
37
For example, you can create methods which can handle any type of members.
Some statically typed languages can do this too. Scala can.

Heard of Psyco? Python code can run in a JIT, which is almost as fast as native C!
It is not. See the benchmarks. Actually it is only slightly fastert than the interpreted Python. The problem is not interpretation - the problem is lack of static types which makes calling methods and accessing fields very costly. Jython is a variant of Python that runs on JVM, and it is JITed. However, it is still much, much slower than Java, Scala or Groovy - because they are statically typed.

See module "re" for regex, "glob" and "fnmatch" for basic wildcard (Unix-style) matching.
Blah. Pattern matching is not textual regular expressions. It is a much more powerful thing. Pattern matching is a general concept common to functional programming languages - it allows you to match objects against object patterns. Not just textual or sequential objects. Used in ML family of languages, Haskell and Scala (and probably some others).

Python will let you write in any whitespace style you like, and frankly, you should be using whitespace anyway.
Ok point taken. However I still don't believe the tabs vs space problem won't bite you at some time.

Nice there are lots of IDEs, but no really working code completion. Most on that list are beta quality.

JPython or Jython lets you run Java libraries from Python and the Python libraries are pretty good in my opinion anyway.
Agreed. Point taken. I forgot about Jython. However, as for JVM languages it doesn't stand a chance with Clojure and Scala. :)
 

rapidcoder

Joined Jan 16, 2011
37
For example, you can create methods which can handle any type of members.
Some statically typed languages can do this too. Scala can.

Heard of Psyco? Python code can run in a JIT, which is almost as fast as native C!
It is not. See the benchmarks. Actually it is only slightly fastert than the interpreted Python. The problem is not interpretation - the problem is lack of static types which makes calling methods and accessing fields very costly. Jython is a variant of Python that runs on JVM, and it is JITed. However, it is still much, much slower than Java, Scala or Groovy - because they are statically typed.

See module "re" for regex, "glob" and "fnmatch" for basic wildcard (Unix-style) matching.
Blah. Pattern matching is not textual regular expressions. It is a much more powerful thing. Pattern matching is a general concept common to functional programming languages - it allows you to match objects against object patterns. Not just textual or sequential objects. Used in ML family of languages, Haskell and Scala (and probably some others).

Python will let you write in any whitespace style you like, and frankly, you should be using whitespace anyway.
Ok point taken. However I still don't believe the tabs vs space problem won't bite you at some time.

Nice there are lots of IDEs, but no really working code completion. Most on that list are beta quality.

JPython or Jython lets you run Java libraries from Python and the Python libraries are pretty good in my opinion anyway.
Agreed. Point taken. :)
 

tom66

Joined May 9, 2009
2,595
Some statically typed languages can do this too. Scala can.
Okay.. Point taken, but there are many other uses for dynamic/"duck" typing.

It is not. See the benchmarks. Actually it is only slightly fastert than the interpreted Python. The problem is not interpretation - the problem is lack of static types which makes calling methods and accessing fields very costly. Jython is a variant of Python that runs on JVM, and it is JITed. However, it is still much, much slower than Java, Scala or Groovy - because they are statically typed.
It depends on exactly what you are doing. If you're doing algorithmic stuff, you can get very high performance. If you're doing lots of method calls, and mixing types, it will be slow.

Blah. Pattern matching is not textual regular expressions. It is a much more powerful thing. Pattern matching is a general concept common to functional programming languages - it allows you to match objects against object patterns. Not just textual or sequential objects. Used in ML family of languages, Haskell and Scala (and probably some others).
I must admit I have never heard of this before. I have no idea if there are Python libraries or native support for it.

Ok point taken. However I still don't believe the tabs vs space problem won't bite you at some time.
Stick with spaces and there will be no problems. In many IDEs, it's possible to get the editor to replace tabs with a number of spaces.

Nice there are lots of IDEs, but no really working code completion. Most on that list are beta quality.
IDLE has code completion, so does NetBeans, as does PyDev, so does WinGlde... in fact, most on that list do! However, I have never used code completion and never found a use for it.

Agreed. Point taken. I forgot about Jython. However, as for JVM languages it doesn't stand a chance with Clojure and Scala. :)
I have nothing to add as I cannot compare languages I have not used - I am only raising points about my personal favourite.
 

rapidcoder

Joined Jan 16, 2011
37
Okay.. Point taken, but there are many other uses for dynamic/"duck" typing.
And what prohibits you from having both static typing and duck typing in the same language? Google Go and Scala are languages that show it is possible. Google Go has exact duck typing, as in dynamic languages, Scala has an enhanced, type safe version of duck typing, called structural typing. Both can do whatever is possible for dynamic languages.

It depends on exactly what you are doing. If you're doing algorithmic stuff, you can get very high performance. If you're doing lots of method calls, and mixing types, it will be slow.
Doing complex algorithmic stuff (like creating a simulation engine) means usually doing lots of method calls, creating lots of classes and objects, etc. It is easy to go high-performance in a microbenchmark, it is not easy in a large application if method calls - a basic glue - are slow and not inlined.

IDLE has code completion, so does NetBeans, as does PyDev, so does WinGlde... in fact, most on that list do! However, I have never used code completion and never found a use for it.
There is a big difference between having it on the list, and actually having it working dependably. I have friends who write lots of Python, and they also don't use code completion - because it almost never works, except in trivial cases, where it is not needed. Code completion is great, especially if you work with code that you haven't written. Saves lots of documentation lookups. Types are documentation, a kind of documentation that is always up to date, so very very valuable.

Someone created a very good blog post on type systems:
http://www.pphsg.org/cdsmith/types.html
 

tom66

Joined May 9, 2009
2,595
And what prohibits you from having both static typing and duck typing in the same language? Google Go and Scala are languages that show it is possible. Google Go has exact duck typing, as in dynamic languages, Scala has an enhanced, type safe version of duck typing, called structural typing. Both can do whatever is possible for dynamic languages.
This is interesting, I found this on Wikipedia (not sure if it is related):

An oft-cited criticism is this: One issue with duck typing is that it forces the programmer to have a much wider understanding of the code he or she is working with at any given time. In a strongly and statically typed language that uses type hierarchies and parameter type checking, it's much harder to supply an unexpected object type to a class. For instance, in Python, you could easily create a class called Wine, which expects a class implementing the "press" attribute as an ingredient. However, a class called Trousers might also implement the press() method. With Duck Typing, in order to prevent strange, hard-to-detect errors, the developer needs to be aware of each potential use of the method "press", even when it's conceptually unrelated to what he or she is working on. In essence, the problem is that, "if it walks like a duck and quacks like a duck", it could be a dragon doing a duck impersonation. You may not always want to let dragons into a pond, even if they can impersonate a duck.

Proponents of duck typing, such as Guido van Rossum, argue that the issue is handled by testing, and the necessary knowledge of the codebase required to maintain it.
Personally, I do not understand structural typing, I need to take a closer look at it.

Doing complex algorithmic stuff (like creating a simulation engine) means usually doing lots of method calls, creating lots of classes and objects, etc. It is easy to go high-performance in a microbenchmark, it is not easy in a large application if method calls - a basic glue - are slow and not inlined.
I'm thinking more like sequential expressions - for example, in one project in Python there were about 20 lines of code which manipulated a matrix, and despite using classes, it did speed up considerably with Psyco (probably around 10x, but I didn't record it.) And I made a program which did a lot of looping - typically something that should be slow - and it was about 20x faster with Psyco.

There is a big difference between having it on the list, and actually having it working dependably. I have friends who write lots of Python, and they also don't use code completion - because it almost never works, except in trivial cases, where it is not needed. Code completion is great, especially if you work with code that you haven't written. Saves lots of documentation lookups. Types are documentation, a kind of documentation that is always up to date, so very very valuable.
As I said - I do not use it and have never found a use for it in any language. So I can't offer much comparsion.
 

tomson

Joined Sep 6, 2010
16
as for code completion - I'm using PyCharm (it's not free though) and it works just fine. comodo also looks nice but haven't tried it.

I agree to most of the points apart of performance - you want performance, than stay away from anything related to Java...still.
 

rapidcoder

Joined Jan 16, 2011
37
Can you back up your opinion with any code or real-world example?

I can see lots of evidence Java's performance is on par with C now, if you take enough large application and exclude stupid microbenchmarks of the type "Look ma, I worked on these 10 lines of C code for a week, it is a terribly unreadable mess now, but works 10% faster than Java".

Some real-world examples:
- Facebook decided to write their high performance database system in Java.
- Amazon also uses pure Java solution for their core database.
- Twitter created their message routing backend in Scala (JVM language), mainly due to performance and scalability considerations.
- Jake2, a Quake2 port to Java is just as fast as the original
- Java Hadoop won some high performance computing competitions, outperforming software written in C++
- Tomcat is just as fast as the Apache webserver, Netty is very close to Nginx.
- HSQLDB or H2 database engine compared to MySQL and PostgreSQL. The C/C++ databases are definitely slower than their Java counterparts.

And one more, taken from my personal experience:

We've recently performance tested our circuit simulation engine prototype running on JVM doing some RLC filter simulation and we found it 2-3 times faster than CircuitLogix, TopSpice and Tina (all written in C). It is still a little slower than LTSpice, however we are still in the alpha stage - the project is only a half year old. I think we are able to match LTSpice in performance soon - a 2x performance improvement is easily possible, but not our priority now. BTW: LTSpice doesn't guarantee global convergence for non-linear circuits, while ours does. Doing things well is more important than doing things fast. :p

The problem with writing big and fast software in C or C++ is that you can either go big, or go fast, but not both, unless you have unlimited budget and time. So while it is easy to outperform Java in a small microbenchmark, it is much harder to do it for a real world big application. Simply, when you get big enough, you have little time left for optimisation - most of the time is spent creating features and fixing bugs. And at this Java is much better than C family languages, especially if we are talking about hard, complex scientific code.

It is <snip> easier to implement good and complex algorithms in Java, Scala or Python than it is in C++. C++ programmers struggle often to get just ANY solution of their problem, because the language adds too much own complexity to it. When moved to a more expressive language, the problem seems simpler and now you can not only think of ANY solution, but also of a FAST solution and optimal algorithm. This is the case with our simulator and SPICE. Ours is not faster because we have implemented the same SPICE algorithms better in Java. We have just used different, better algorithms.
 
Last edited by a moderator:

cj3

Joined Mar 9, 2012
11
The problem with Python is two-fold: 1. Its syntax (obfuscated C); and 2. Weak typing. A better thought out syntax is Ada; and strong typing should be forced, such as in Ada or TrueBASIC.
 

ingframin

Joined Feb 2, 2012
1
Python and Java are different languages with different purposes.
In an electronic lab the biggest programming activity is scripting to automate measurements or analyzing data.
Python for this is perfect because Java requires a good object oriented design in order not to have terrible code and it's not suited for scripting.
Consider also that loading in Java the VISA driver or the Daqmx driver is really complicated...
With Python this is easy and we can go way faster in developing our scripts than with Java.
I love Java, but I use it for other purposes :)
I think it's a shared opinion that the best is to know both languages (and possibly some other language) in order to make the best choice depending on the case.
About speed problem... I must admit that usually the main speed problem is between the chair and the keyboard ;) (but mobbing my friends that are computer scientists I can solve a big part of this issues :p )
 

takao21203

Joined Apr 28, 2012
3,702
The problem with Python is two-fold: 1. Its syntax (obfuscated C);
That is what I was thinking when I examined it. Some kind of weird C. Same with PHP, and Ruby. Yes I know these are interpreted, dynamic, scripted languages.

I found it difficult enough to keep up with these new C++ extensions, and at the same time, also to use embedded C.

If you look at websites these days, they are so full of unneccessary things...for sure, hordes of script language programmers are employed, because it is so difficult to use these languages.

I don't really understand what is so new or revolutionary with these script languages.
 

KL7AJ

Joined Nov 4, 2008
2,229
A week or two ago a discussion in another thread made me think about writing a document as to why the programming language python and its add-ons are good tools for a technical person to have in his toolkit. I'm not suggesting it should take the place of other languages (indeed, an experienced technical person will know many languages), but I've found it an excellent general-purpose programming tool and it is usually what I turn to first.

I wrote the attached document to introduce some of the features of the language and discuss some of the add-ons that provide extra power. You can also get the zip file here (see the bottom of the page).

Edit 17 Nov 2010: I updated the document to add another example and discuss the problem of choosing between python 2.x and 3.x in more detail. The latest version will always be here: http://code.google.com/p/hobbyutil/downloads/list. I recommend getting the file from this link rather than using the file attached to this post.
 
Top