Posted on March 16, 2008 00:34 by jpager

Two days ago, I posted an example of a way to apply IronPython scripting in .NET. At the very end of my post, I wondered about the possibility of compiling the script as a .NET assembly.

I'm happy to say, I have found a way to implement this functionality (and I might mention at this point that this is my last post about IronPython). First, I'll explain how I was able to do it, and then I'll explore whether we really gain anything from having a pre-compiled script.

First, compiling the script. The idea is that we only want to recompile our script if (a) the DLL does not exist or (b) our script is more recent than the DLL (i.e., it has been edited). We'll examine our files using the System.IO.FileInfo class. If we need to compile the script, we'll use IronPython's implementation of System.CodeDom.Compiler.CodeDomProvider, which will allow us to compile a python script into MSIL, and save the assembly as a DLL (this is located in the IronPython.CodeDom namespace):

FileInfo fi_in = new FileInfo(exportScript);
FileInfo fi_out = new FileInfo("BatchExport.dll");
if (!fi_out.Exists || (fi_in.LastWriteTimeUtc >= fi_out.LastWriteTimeUtc))
{
   PythonProvider p = new PythonProvider();
   CompilerParameters _params = new CompilerParameters(new string[] { }, "BatchExport.dll");
   CompilerResults r = p.CompileAssemblyFromFile(_params, new string[] { exportScript });
   if (r.Errors.Count > 0) throw new Exception("Compiler error.");
}


This is fairly straightforward. We create a compiler, call CompileAssemblyFromFile(), which is configured to save the assembly as BatchExport.dll and compile the file containing our python script. The compilation takes a short amount of time, so we only want to compile if absolutely necessary.

The next step is actually calling into the compiled assembly. As I mentioned last time, calling a Python assembly from C# is non-trivial. That python assembly contains dynamic types, which are not easily reflected into useful C# objects. So my approach was to create a PythonEngine, import the assembly into said PythonEngine, and call the method using Python.

To do this, I first altered my script so that my Export() method is contained in a class called BatchExport, rather than being a global method. Then, in my C# code, I created an instance of the BatchExport class, and called the Export() method on that object.

using (PythonEngine engine = new PythonEngine())
{
   engine.AddToPath(AppDomain.CurrentDomain.BaseDirectory);
   engine.Import("clr");
   engine.Execute("clr.AddReference('BatchExport.dll')");
   engine.Execute("import BatchExport");
   engine.Execute("be = BatchExport()");
   List<string> _params = new List<string>();
   _params.Add("order");
   ExportDelegate exMethod = engine.CreateMethod<ExportDelegate>("return be.Export(order)", _params);
   bool res = exMethod(order);
}


Our first step was to actually load our assembly into our python engine. To do that, we call clr.AddReference(). This allows us to include any .NET assembly in our script.

Then, we instantiated our BatchExport object, and use the CreateMethod() method to actually call the object's Export() method. Now the compile-script-only-if-it-has-changed functionality works perfectly.

Of course, after doing this, I made a big mistake--I decided to see how much time this actually buys us. So I created a simple script that processes a batch 500 times using the old method (that simply calls ExecuteFile()) and the same batch 500 times using the new method (I also created a new Export() python script that doesn't write to any files, in order to avoid any bias involving file sizes). The result was at least a little surprising--it took the method that pre-compiles the script 15 times as long to process the batches as it does the method that executes the script anew each time.

Possible culprits? Probably the cost of reflection is the main one. I don't think the check on the file times could make that big a difference. Clearly this little operation is quite expensive:

engine.Import("clr");
engine.Execute("clr.AddReference('BatchExport.dll')");
engine.Execute("import BatchExport");

So, I guess what we can take out of this is that compiling python scripts to .NET DLLs comes at a considerable cost, both in performance and complexity, at least if the script is simple. It may be cheaper to precompile the assembly if our script is hundreds (or thousands) of lines long, but for fairly basic scripts, you might as well just leave them as scripts.



Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Posted on March 14, 2008 01:07 by jpager

Last week I explored the PythonEngine object in IronPython. We didn't get any deeper than simply using the API--this week I figured I'd actually put it to use in a real application.

Here is our scenario. You are a developer for a software company that writes a Point of Sale application on the .NET platform. You have hundreds of clients, each with their own in-house IT departments. You'll install everything for them, provide some support when needed, but you simply don't have time to offer each client major customizations. The problem is, each client has its own billing system--some legacy, some more modern--and there is no way for your development team to write a module to interface with each individual system. So, our approach will be to use scripting for the export module. This will allow our clients' in-house development teams to make changes to the export module on their own (or even rewrite the whole thing to their hearts' desire!) without having to even recompile the application. And, you guessed it, we'll use Python for our scripting (the solution and the scripts are attached).

Our first step is to actually build our point of sale application. Sparing all the technical details, I've created a simple Windows Form with a list of products, some fields for quantity, totals, etc., and a "Process" button. When the button is clicked, it will use our python script to send the order out to the billing system. To make things even more customizable, I'll also create an app.config file to store the location of the export python script.

Actually loading the python script is trivial--even more trivial than anything we discussed last week. We need nothing more than a call to the PythonEngine's ExecuteFile() method:

public static void Process(Batch order)
{
   // Do some stuff... Save to database, logs, etc.
   // Send outbound
   string exportScript = ConfigurationManager.AppSettings["ExportScript"];
   using (PythonEngine engine = new PythonEngine())
   {
      engine.ExecuteFile(exportScript);
      bool res = (bool)((IronPython.Runtime.Calls.PythonFunction)engine
         .Globals["Export"]).Call(order);
   }
}

And viola, we're finished with our C# code. We can package this and send it our for our clients. Just to be nice, we'll also provide two simple export scripts. One will be a flat file format that will be dropped off for a legacy system to pick up, and the other will generate simple XML for a more modern system.

Here is our first script:

def Export(batch):
   f = file('export.txt', 'a')
   f.write('BATCH BEGIN [[')
   f.write(str(batch.SubTotal) + '|' + str(batch.Tax) + '|' + str(batch.Total) + '\r\n')
   for item in batch.Items:
      f.write(item.Description + '\r\n')
      f.write('BATCH END ]]\r\n')
   f.close()
   return True

When we run this a couple of times, we get the following output:

BATCH BEGIN [[1.0|0.0850|1.0850 
Mountain Dew 
Mountain Dew 
BATCH END ]] 
BATCH BEGIN [[56.99|4.84415|61.83415 
Mountain Dew 
Mountain Dew 
Arrested Development (DVD) (entire series) 
BATCH END ]]   

And we'll also provide a script that exports it in a simple XML format:

def endl(file):
   file.write('\r\n')
def Export(batch):
   f = file('export.xml', 'a')
   f.write('<batch ')
   f.write('subtotal="'+str(batch.SubTotal)+'" ')
   f.write('tax="'+str(batch.Tax)+'" ')
   f.write('total="'+str(batch.Total)+'">')
   endl(f)
   if(len(batch.Items) == 0):
      f.write('\t<items />')
      endl(f)
   else:
      f.write('\t<items>')
      endl(f)
      for item in batch.Items:
         f.write('\t\t<item description="'+item.Description+'" />')
         endl(f)
      f.write('\t</items>')
      endl(f)
   f.write('</batch>')
   endl(f)
   f.close()
   return True  

This gives us the following output:

<batch subtotal="42.0" tax="3.5700" total="45.5700">
   <items>
       <item description="DVD Player" />
      <item description="Novelty Frisbee" />
      <item description="Novelty Frisbee" />
      <item description="Novelty Frisbee" />
      <item description="Novelty Frisbee" />
   </items>
</batch>
<batch subtotal="19.75" tax="1.67875" total="21.42875">
   <items>
      <item description="Novelty Frisbee" />
      <item description="Novelty Frisbee" />
      <item description="Novelty Frisbee" />
      <item description="Nebraska Replica License Plate" />
      <item description="Erick Dampier jersey (XXL)" />
      <item description="Erick Dampier jersey (XXL)" />
   </items>
</batch>    

And these are just two of the possibilities. Our clients can interact with their external system in any way imaginable--TCP, Message Queue, Web Service, Database--anything that is possible with .NET or python can be done.

Our last consideration is one of efficiency. It would be nice if we could save our compiled script in an assembly and use the pre-compiled version as long as the script is not updated. IronPython allows us to easily create a DLL with our python script using the IronPython.CodeDom namespace, but it is difficult to invoke the compiled code in C#, if it is even possible. A possible workaround may be to write a python script that references the assembly and invoke that script instead. I'll let you know if I ever get that to work.

PythonInterface.zip (15.66 kb)



Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Posted on March 10, 2008 14:45 by swilliams

Recently stumbled across some pretty neat implementations of the [in]famous FizzBuzz question in one line. I saw a python based solution, but felt it could be tweaked a little bit more:

print "\n".join(["%s" % ['fizzbuzz','fizz','buzz',i][[i%15,i%3,i%5,0].index(0)] for i in range(1,101)])

This is very clever, and not terribly hard to understand once you know the syntax. Let's break it down into its parts.

First, is the "\n".join(). Python has a join method on strings that inserts the string as the delineator of a list. Coming from a C#/Java background this seems backwards, but hey, it works.

Moving along is the list that is joined by the newline. ["%s" % expression for i in range(1,101)] means that a list of strings will be created for each integer from 1 to 101, and the value will be determined by 'expression'.

['fizzbuzz','fizz','buzz',i] is the list of the potential objects to print, according to the rules of FizzBuzz. i is the integer that will be printed if we are not going to print the other three strings.

[[i%15,i%3,i%5,0].index(0)] is where the decision to print a word is made. It creates a list on the fly to see if the current integer is a multiple of 15, 3, or 5. index(0) then gets the first position that '0' occurs. This tells the prior list which index to use. So:

When i = 1, [i%15,i%3,i%5,0].index(0) becomes [1,1,1,0].index(0), which will return 3. Then ['fizzbuzz','fizz','buzz',i][3] is 'i', which in turn prints "1".

When i = 3, [i%15,i%3,i%5,0].index(0) becomes [3,0,3,0].index(0), which will return 1 (the first position of '0'). Then ['fizzbuzz','fizz','buzz',i][1] is 'fizz', which will be printed.

What I like about this is that it is a "legitimate" one liner. Yes, you could cook up something similar in C# with gratuitous semicolons, but that violates the essence of style that C# has.


Python Bites are little bits of Python code that have helped me out in my day to day. I wouldn't necessarily build an enterprise level application out of it, but it is usually quicker for smaller scripting tasks.



Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Posted on March 7, 2008 22:44 by jpager

I don't know about you, but I am a fan of python. It has the advantages of flexibility that come from weakly typed languages, but maintains many of the functions of typical object-oriented languages. Though it feels like a scripting language, you can even write fully-featured standalone applications if you feel so inclined. So, naturally, I thought it would be fun to explore the possibility of using Python with .NET.

The de-facto standard .NET implementation of Python is IronPython. IronPython is a Python implementation that is written in .NET, and has use of all of the .NET CLR functionality. This allows us a couple of possibilities. We can use it as a python interpreter that has access to .NET functionality (for example, Windows Forms). However, as Scott has pointed out, if you're looking for a Python interpreter, you might as well use the native one, and if you're writing a .NET application, you might as well use C#. The other way we can use IronPython is purely as a scripting language that can be embedded in a C# (or VB, if you swing that way) application. (Particular ways to use scripting in .NET, while perhaps the most practical aspect of this discussion, will be reserved for some other time. My focus will be nothing more than using IronPython in a C# application.) So, without further ado...

Using IronPython in C#

At the center of the IronPython module is the PythonEngine object. So, the first thing we need to do is import our IronPython dll and create our PythonEngine object. Let's start by importing the IronPython.Hosting namespace:

using IronPython.Hosting;  

We create our object like so:

     using (PythonEngine engine = new PythonEngine()) {
     }  

Now that we have our PythonEngine, we can explore some of its basic functions.

The most basic function of PythonEngine is the Evaluate() method. Evaluate simple executes a single Python expression (note: by this I really mean expression, not statement. If you try an assignment, it will complain about bad python syntax.) and returns the result as an object (there is also a strongly-typed version, EvaluateAs, which I will use in my examples).

Let's try approximating PI using a python expression:

     double pi = engine.EvaluateAs<double>("4/1.0 - 4/3.0 + 4/5.0 - 4/7.0 + 4/9.0 "
          + "- 4/11.0 + 4/13.0 - 4/15.0 + 4/17.0 - 4/19.0 + 4/21.0 - 4/23.0 + 4/25.0 "
          + "- 4/27.0 + 4/29.0 - 4/31.0 + 4/33.0 - 4/35.0 + 4/37.0 - 4/39.0 + 4/41.0 "
          + "- 4/43.0 + 4/45.0 - 4/47.0 + 4/49.0 - 4/51.0");   

When we check the value of our pi variable, we get the value 3.1031453128860127, just as we expect.

Obviously, our value of pi is a little off, so we probably want to use the value of pi in python's math module. So, let's change our expression to this:

     double pi2 = engine.EvaluateAs<double>("math.pi"); 

If you try this, you'll get a runtime error: PythonNameErrorException: name 'math' not defined (and yes, the Exception class is "PythonNameErrorException"). This brings us to our next feature of the PythonEngine: Import. To get this to work, we need to import the math module, like so:

     engine.Import("math");
     double pi2 = engine.EvaluateAs<double>("math.pi"); 

Now, as we would expect, our pi2 variable now contains the value 3.1415926535897931. If you want to use your native python implementation's modules, you need to add those modules' path to your PythonEngine before it could find them:

     engine.AddToPath("C:\\Python25\\Lib");  

Let's move on to something a little more complicated: CreateMethod<TDelegate>(). CreateMethod is a strongly-typed function that returns a delegate. So, obviously, we need to define our delegate, with a contract that the function must adhere to:

     public delegate string DoSomethingDelegate (string param); 

CreateMethod has four overloads, but the one we'll deal with is the one that takes a python statement as the first parameter and a list of type names (the function signature) as the second:

     List<string> _params = new List<string>(new string[]{"param"});  
     DoSomethingDelegate function = engine.CreateMethod<DoSomethingDelegate>(@"
return 'Did you say \'' + param + '?\''  
          ", _params); 
     string rtn = function("TEST"); 

The last feature we'll look at is PythonEngine's Execute method. Execute is pretty self-explanatory--it simply executes Python code. The difference between Execute and the previous functions we've discussed is that Execute does not return anything--it only has side effects. It is important to understand that the PythonEngine works similarly to a native Python shell: any names that are defined while executing code are stored in the engine for future use. That is, if you define a name when you call Execute(), that name will be available to every subsequent Execute() call. Here is a simple example:

      engine.Execute(@"
def Foo():
     return 'Bar'
          ");
     string foo = engine.EvaluateAs<string>("Foo()"); 

In our execute() call, we defined a function called Foo(), and then referenced that same function when we called EvaluateAs().

Next, we'll actually call one of our Python functions from C# code. All of the global names in our PythonEngine are publicly accessible through the PythonEngine's Globals property. Any name that is a function is represented by the C# type IronPython.Runtime.Calls.PythonFunction, which has a method called, aptly enough, Call():

      engine.AddToPath("C:\\Python25\\Lib");
     engine.Import("os");
     engine.Execute(@" 
def RunCommand(command):
     p = os.popen(command)
     output = p.read()
     return output
          ");
     string ipconfig = (string)((IronPython.Runtime.Calls.PythonFunction)engine.Globals["RunCommand"])
          .Call(new object[] { "ipconfig" }); 

This does nothing more than create a process with the command sent through the parameter (which in this case is ipconfig) and read the output stream of the process. Because it uses the native Python os module, I added the path to that module to the PythonEngine's path. After this code executes, the ipconfig variable contains the output of the Windows ipconfig utility.

Now, let's put this to use and create something with it. We'll build a simple web form containing a panel, a text box, and a button. When the button is clicked, the command in the text box will be executed, and the output displayed in the panel (Disclaimer: Whatever you do, do not deploy this page to a network facing web site, for reasons that should be obvious to you).

Here's a snippet of our .aspx page:

     <asp:Panel ID="pnlOutput" runat="server" />
     <asp:TextBox ID="txtCommand" runat="server" Width="600" />
     <asp:Button ID="btSubmit" Text="Execute" runat= server  /> 

Here's our button's event handler:

     void btSubmit_Click(object sender, EventArgs e)
     {
          using (PythonEngine engine = new PythonEngine())
          {
               engine.AddToPath("C:\\Python25\\Lib");
               engine.Import("os");
               engine.Execute(@"
def Execute(command):
     pipe = os.popen(command)
     return pipe.read()
                    ");
               PythonFunction command = (PythonFunction)engine.Globals["Execute"];
               string res = (string)command.Call(new object[] { txtCommand.Text });
               pnlOutput.Controls.Add(new LiteralControl("<pre>" + res + "</pre>"));
          }
     } 

And there you have it. Try it out and play with it; we've only barely touched the possibilities.



Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Posted on March 5, 2008 16:02 by swilliams

I needed to create an easy to use file of state abbreviations and names. I found something close from the USPS, but the format would have been a pain to parse in my application. So I decided to use Python to rearrange the text into something a little more sensible:

f = file('c:\users\swilliams\documents\code\states.txt', 'r')
for line in f:
	print line[-3:-1] + ',' + line[:-3].strip()
f.close()

And it worked perfectly. Not bad for 4 lines of code.


Python Bites are little bits of Python code that have helped me out in my day to day. I wouldn't necessarily build an enterprise level application out of it, but it is usually quicker for smaller scripting tasks.



Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5