Cloud Computing Thoughts

SQL Patch Generator

azuredba@gmail.com — Thu, 23 Jun 2011 01:43:56 GMT

Over the last several months, I’ve had to write patch scripts for database updates. Writing the scripts tends to be easy and I came up with a fairly effective way to implement changes but writing the actual script got to be tedious. So, like any good programmer, I automated the task.

My Patch Implementation

There are a lot of ways to write patch scripts, but one main concern is that you don’t overwrite procedures with old versions. For my environment, I came up with a fairly simple solution for procedures and user defined functions. Every update to a procedure or UDF creates a time stamped named version of itself. If I was generating a patched version of auditing.GeneralLogInsert today, I would check to see if auditing.[GeneralLogInsert-2011-06-22] exists. If it does, then I don’t proceed any further with that stored procedure. If it doesn’t, then I populate an nvarchar(max) variable with the create statement for the procedure/UDF, execute a rename on the existing object and then execute the create. It looks something like this:

IF NOT EXISTS(SELECT * FROM sys.all_objects WHERE name = GeneralLogInsert-2011-06-22' and schema_name(schema_id) = 'auditing')
BEGIN
declare @SQLCommand nvarchar(max)
SET @SQLCommand = '-- =============================================
-- Author:        Josef Finsel
-- Create date: 2010-11-20
-- Description:    Record Log Information
-- =============================================
CREATE PROCEDURE auditing.GeneralLogInsert
           (@DomainName nvarchar(50)
           ,@FolderPath nvarchar(2083)
           ,@PageName nvarchar(2083)
           ,@Method nvarchar(10)
           ,@IPAddress nvarchar(50)
           ,@UserID nvarchar(255)
           ,@LogEvent nvarchar(100)
           ,@LogMessage nvarchar(max)
           ,@ParameterInformation xml)
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;
declare @InternalUserID bigint
INSERT INTO [auditing].[GeneralLog]
           ([DateLogged]
           ,[DomainName]
           ,[FolderPath]
           ,[PageName]
           ,[Method]
           ,[IPAddress]
           ,InternalUserID
           ,[LogEvent]
           ,[LogMessage]
           ,[ParameterInformation])
     VALUES
           (getdate()
           ,@DomainName
           ,@FolderPath
           ,@PageName
           ,@Method
           ,@IPAddress
           ,@InternalUserID
           ,@LogEvent
           ,@LogMessage
           ,@ParameterInformation)
END
'
exec sp_rename 'auditing.GeneralLogInsert, GeneralLogInsert-2011-06-22', 'object';
EXEC (@SQLCommand)
END
GO

To generate a script like this, I used to have to go into SSMS, right click on the object and script as create to new query window, remove the header information, change all sinqle quotes to double single quotes and then put it into the script. On more than one occasion I found myself having to fix scripts simply because I didn’t match the object names up correctly. Now I don’t have to do this, I can use my utility to create the script block for me.

Running the Utility

The heart of the utility is actually in a class that can be expanded on and used in a number of places, but I have put the class inside an EXE that can be called. From there I can redirect the output into a file.

GenerateSQLPatch.exe -? -c[onnection]:dbConnection -s[chema]:SchemaName -o[bjectname]:ObjectName -update
-? shows this message
-c[onnection] the connection string used to connect to the server
-s[chema] the schema name of the object
-o[bjectname] name of the object to be created)
-update Flag to indicate whether this is a new object or update of an existing

Get the Code

I’ve posted the code in an open project on GitHub. You can find it here. Feel free to modify it to suit your needs and to offer up contributions for improving it.

Technorati Tags: SQL Server,Patching,Utilities

Life Cycle of Coding

azuredba@gmail.com — Mon, 25 Apr 2011 20:27:00 GMT

Build
Debug
Learn your Mistakes during Maintenance
Code Around your Mistakes
Go to Version +1 and try to undo your mistakes without being able to start with a clean slate and make a new set of mistakes while cringing at all you didn't know that you didn't know not realizing how much you don't know now that you'll regret later
Repeat as necessary

Implementing Search in Windows Azure

azuredba@gmail.com — Tue, 18 Jan 2011 02:11:22 GMT

It’s been a couple of months since I last posted about the MOOP Framework, but we’ve been working on some exciting new things. Today we’re going to talk about implementing Search in Azure using Lucene. Actually, we’re using a modified version called Lucene.Net and the Azure Library for Lucene.NET.

Implementing search actually breaks down into two separate tasks. The first is indexing your data and the second is delivering up search results. Let’s look at each of these in turn.

Indexing Data for Search

The easiest way to do this is to implement a worker role. On the negative side, this means I now am paying an additional chunk of money for a processor but it has a lot of advantages that make it worth the extra cost. Our worker role really just checks to see if there are any messages in queue, processes them and then sleeps for 10 seconds before trying again. For simplicity’s sake, I’m just showing the part of the code that processes the message.

if (messageData != string.Empty)
{
  string queryName = string.Empty;
  NameValueCollection htKeys = new NameValueCollection();
  foreach (string messageDetail in messageData.Split("\r\n".ToCharArray()))
  {
    if (messageDetail.Contains(":"))
      htKeys.Add(messageDetail.Substring(0, messageDetail.IndexOf(":")), messageDetail.Substring(messageDetail.IndexOf(":") + 1));
    else
      if (messageDetail != string.Empty)
        queryName = messageDetail;
  }
  quoteSearchLoader qls = new quoteSearchLoader();
  if (htKeys.Count == 0)
    aqs.Messages(cmdType.delete, queueName, string.Empty, string.Format("popreceipt={0}", popReceipt), messageID);
  qls.processRequest(queryName, htKeys);
  if (htKeys.Count != 0)
    aqs.Messages(cmdType.delete, queueName, string.Empty, string.Format("popreceipt={0}", popReceipt), messageID);
}
else
  aqs.Messages(cmdType.delete, queueName, string.Empty, string.Format("popreceipt={0}", popReceipt), messageID);
}

The most interesting thing here is how I package up the data in the message. The first line is basically a pointer to a blob that contains the information needed to build the index. Any other lines contain parameters that can be passed into the database. Let’s take a look at both a message

oneQuote
QuotePublicID:EA329A1E-179B-4890-ABEB-4C9F94638C50

And now the XML that is paired with that message:

If you think that the stored procedure section looks familiar, you're right. That's the same style of XML we used for our data template before. Since we’ll be doing the same thing, there’s no reason to reinvent the wheel. The XML above is actually stored in a blob called oneQuote.xml. So, as you can see, the first line of our message defines which blob to use. And our stored procedure has one parameter, QuotePublicID, which is on the second line of our message as a name value pair separated by a colon. The Worker class just takes the message queue, parses it out and passes the name of the query and the NVP table to the Quote Search Loader class, which is where the fun begins.

Quote Search Loader

The Quote Search Loader class uses the lucenedata section of the XML to determine how to load the data from the database into Lucene. There are a number of good references on how to create your documents in Lucene if you search Google. This is more about how we go about indexing our data without recompiling. To do this, we have a definition of the fields that we are going to create. Each field has several attributes:

dataname is the name of the field in the data set retrieved with the stored procedure call
lucenename is the name of the field in the document that we are going to create
index is a text representation of Lucene.Net.Documents.Field.Index and determines how the document will index this field for retrieval.
store is a text representation of Lucene.Net.Documents.Field.Store and determines whether the data will be stored in the document so it can be presented in the search results

Now, the table of quotes that I have contain two fields that I want indexed but don’t want to store, approved and currentStatus. If a quote has not been approved then I don’t want it appearing in the results. All of the other fields I want to have available to show in the output.

So, the first thing the QuoteSearchLoader does is to parse the lucenedata elements into a set of arrays. Then it parses the storedprocedure data and makes the database call. Once it has data, it walks through the returned data set

while (dr.Read())
{
  StringBuilder olioSearch = new StringBuilder();
    Document doc = new Document();
  for (int i = 0; i <= dataName.GetUpperBound(0); i++)
  {
    if(i==0)
    {
      idxW.DeleteDocuments(new Term(luceneName[i], dr[dataName[i]].ToString().ToLower()));
      doc.Add(new Field(luceneName[i], dr[dataName[i]].ToString().ToLower(),Field.Store.YES, Field.Index.NOT_ANALYZED ));
    }
    else
      try
      {
        doc.Add(new Field(luceneName[i], dr[dataName[i]].ToString(), fieldStore[i], indexType[i]));
       if ( indexType[i] == Field.Index.TOKENIZED || indexType[i] == Field.Index.ANALYZED)
          olioSearch.AppendFormat("\r\n{0}", dr[dataName[i]].ToString());
      }
      catch (Exception ex)
      {
        NotifyError(ex);
      }
  }
  if(olioSearch.ToString() != string.Empty)
    doc.Add(new Field("olioSearch",olioSearch.ToString(), Field.Store.NO, Field.Index.ANALYZED_NO_NORMS));
  idxW.AddDocument(doc);
}

There are only a couple of interesting things here. First, the first item in the LuceneData block is used as the key to be able to uniquely identify the object. This allows me to delete the existing document rather than having multiple copies of it. I probably should modify this to be an attribute somewhere but that’s an optimization for later. With the data I loop through the array of attributes and create my document by adding fields as defined by the XML.

The second thing is that every column that’s defined as analyzable is being added to a generic field in the document called olioSearch. For my purposes, having one field that contains all of the searchable text makes sense so that a search of WILD will return not only quotes with the word wild but also those by Oscar Wilde. It’s another item that should be an attribute on the lucenedata/field element but that will be for the next revision. With this in place, I can add new data to be indexed without having to recompile, just define the database stored procedure and the XML file that defines how to index the data.

LuceneSearch Handler

Now that I have data indexed, I need a way to get search results. For this, I have created a generic handler and created a definition in my web.config to point all pages that have the ls (lucene search) extension to this handler. Just as I have done with data access, the page name is used to get an item out of blob storage that defines how to search and how to represent the results. Let’s look at a sample control XML for searching quotes:



  
    searchQuotes.xslt

The luceneSearch element has two child elements: transform, containing an XSLT transform to apply to the search results and query, which defines how to build the search query.

The query elements help shape the search. The simplest to explain are those with a variableValue of false. These are simply applied as boolean queries with the value in termValue applied to the field name in term. Every search that gets made with this control xml will return indexed documents with approved=Y and currentstatus = 0. When variableValue is true, however, the termValue is used to get a value from the parameter list. The beauty of this is that http://azurearchitect.cloudapp.net/quoteSearch.ls?query=truth returns a list of those documents with Truth in the olioSearch field. Adding &approved=n will still return the same values because the only parameter value that will ever be used in this control is query.

Now, let’s look at the handler code.

var q = new BooleanQuery();
XmlNode xn = xdoc.SelectSingleNode("//luceneSearch[1]");
XmlNodeList xnl = xn.SelectNodes("//query");
Query[] qArray = new Query[xnl.Count];

for (int i = xnl.Count - 1; i >= 0; i--)
{
  XmlNode node = xnl[i];
  string term = node.Attributes["term"].Value;
  string termValue = node.Attributes["termValue"].Value;
  string variableValue = node.Attributes["variableValue"].Value;
  if (variableValue == "true")
    termValue = context.Request.Params[termValue].Replace("+"," ").Replace("%20"," ");
  if (i == 0)
    qArray[i] = new QueryParser(Lucene.Net.Util.Version.LUCENE_29, term, new SnowballAnalyzer("English")).Parse(termValue);
  else
    qArray[i] = new TermQuery(new Term(term, termValue));
  switch (node.Attributes["booleanClauseOccurs"].Value.ToLower())
  {
    case "must": q.Add(new BooleanClause(qArray[i], BooleanClause.Occur.MUST)); break;
    case "must_not": q.Add(new BooleanClause(qArray[i], BooleanClause.Occur.MUST_NOT)); break;
    case "should": q.Add(new BooleanClause(qArray[i], BooleanClause.Occur.SHOULD)); break;
    default: q.Add(new BooleanClause(qArray[i], BooleanClause.Occur.MUST)); break;
  }
}
IndexSearcher searcher2 = new IndexSearcher(azDir, true);
TopScoreDocCollector collector = TopScoreDocCollector.create(1000, true);
searcher2.Search(q, collector);
int indexID = 1;
ScoreDoc[] hits = collector.TopDocs().scoreDocs;
StringBuilder xmlOutput = new StringBuilder();
xmlOutput.AppendFormat("");
for (int i = 0; i < hits.Length; ++i)
{
  xmlOutput.AppendFormat("");
  int docId = hits[i].doc;
  Document d = searcher2.Doc(docId);
  xmlOutput.AppendFormat("{0}{1}", hits[i].score, indexID ++);
  foreach (Field f in d.GetFields())
  {
    if (f.StringValue() == null)
      xmlOutput.AppendFormat("<{0} />", f.Name());
    else
      xmlOutput.AppendFormat("<{0}>", f.Name(), f.StringValue());
  }
  xmlOutput.AppendFormat("");
}
xmlOutput.AppendFormat("");
retVal = xmlOutput.ToString();

It's a long snippet but somewhat important. Again, I still have some dependencies that need to be ironed out and cleaned up but it’s good enough for a start.

Whatever the first query item is, that will be run through the analyzer (line 14). Then, depending on the type of booleanClauseOccurs attribute, it’s joined together with the other query items until a query is created, passed in to search and a list of hits is returned. Lines 32 through 48 create an XML document that creates all of the stored fields as elements under a hit element

With this XML, I can easily use a transform to create an HTML page.

When I finished, it only took a couple of minutes to add a new search page by defining a new control XML and a new XSLT, no recompiles necessary.

Technorati Tags: Windows Azure,MOOP Programming Principles,Handlers,Lucene Search

MOOP Framework Data Access Via Handler

azuredba@gmail.com — Mon, 22 Nov 2010 15:19:23 GMT

Today I’m going to delve into how the MOOP Framework handles data access using a combination of a Web Handler and Template Definitions. This handler can get data out of the database, serve it up and even set the cacheability of that data. Let’s begin by looking at the Handler in some detail...

More...

MOOP Framework: Virtualizing Files

azuredba@gmail.com — Tue, 16 Nov 2010 17:36:14 GMT

MOOP stands for Matter Out Of Place, and the MOOP Programming principles can really be summed up as stating that every part of program has a place where it should be running and having everything running in its place provides a built in efficiency that we can harness, an efficiency that’s especially important in cloud computing where we pay for our computer usage.

Yesterday, as I was introducing the MOOP framework, I introduced the concept of Handlers and showed their benefits. Today, we’re going to talk about the first handler that is part of the MOOP Framework, AzureBlobHandler. As the name implies, this handler processes blobs but in doing so, it removes the need for local files from our web role and helps us define caching in a way that drastically reduces our cloud computing costs.

One of the main principles of MOOP Programming Principles is determining where programming should be located and making sure that it is carried out there. Yesterday I explained how this included removing as much of the UI processing as possible from the web server and offloading it onto the client. This means utilizing DHTML as much as possible, which also means that most of what our web server is going to be delivering is static content. The plus side of this is that delivering static files is the simplest task that a web server can do. The downside of delivering static files from the web site is updating them. The easiest choice for delivering static files from an Azure Web Role is to include the files in your web role. The problem of this is that changing these files require a redeployment of the files.

With the MOOP Framework, however, we have an easier way to do it: use Azure Blob Storage. All of the HTML, JavaScript and Images on http://azurearchitect.cloudapp.net/ are being delivered from our Azure Blob Storage. This takes advantage of the fact that there are no bandwidth charges between web roles and Azure Blob Storage, which means it doesn’t cost us anything to have the web role getting information from Blob Storage. In addition, we can then set caching in such a way that static files can be cached publicly, further cutting down on how much bandwidth we incur.

The basic flow of this handler is really very simple. To begin with, it gets the Azure Account information out of the Configuration, meaning changes can be made to the storage with minimal disruptions to the web role. Then it checks to see if the file being requested exists locally. If it does, then we’re going to deliver up the file locally, but we need to take care of some housekeeping to implement caching and cut down on future requests if the client already has a copy of the file. To do this, we need to generate an ETag. A good ETag will be unique not just to the file but to this version of the file. To this end, we create an ETag using the MD5 hash of the file’s last write time and the MD5 hash of the physical file location. Given that these two values should be the same for all instances, we should be good.

With the ETag generated, now we check to see if the browser has included an ETag in the headers collection. If it has, and if the ETag matches, then we can set the HTTPResponse.StatusCode to NotModified, which tells the client that it can use the cached copy. If the ETags don’t match (or if there is no ETag in the headers collection) then we set an Expiration time of one hour and state that the file can be publicly cached, get the MimeType of the file and send it off to the client.

If, on the other hand, the file doesn’t exist locally, we assume that the file is going to be one we can get out of Blob Storage. We use the path of the URI to determine the location of the object in storage, so http://azurearchitect.cloudapp.net/scripts/imgs/color.png would be the blob imgs/color.png in the scripts container. If the request contains an ETag and that ETag matches the version in Blob Storage, we set the HTTPResponse.StatusCode to NotModified and we’re done. If the ETag doesn’t match or there is no ETag, we get the item out of Blob Storage, set the ETag, set the Cache Expiration and set caching to Public and send the item off. If the item doesn’t exist, we pass the 404 back that Azure Blob Storage returns to us.

If you’re interested in getting our MOOP Framework, please contact us and we’ll be happy to discuss how we can help you.

Tomorrow we’ll post about how to use a Handler to interface with your data and some of the fun experiences that IE brings to AJAX Queries.

Technorati Tags: Windows Azure,MOOP Programming Principles,Handlers

Introducing the MOOP Framework for Azure

azuredba@gmail.com — Mon, 15 Nov 2010 15:49:44 GMT

MOOP stands for Matter Out Of Place, and the MOOP Programming principles can really be summed up as stating that every part of program has a place where it should be running and having everything running in its place provides a built in efficiency that we can harness, an efficiency that’s especially important in cloud computing where we pay for our computer usage.

To introduce the MOOP Framework, I’m going to talk about a new implementation of the AzureArchitect Web Tools, this one using HTML, JavaScript and Handlers. The old version of the tools would allow you to enter your Account and SharedKey and then work with Blobs, Queues and Tables. This was handled by several different ASPX pages, one for each type of Azure Storage. Let’s take a look at the Blob Storage ASPX page as an example.

Once you’ve filled in your account and shared key, you could get a list of containers, for instance. Every time the button is pressed to get a list of containers, a call is made to the ASPX, which then uses the AzureCommands DLL to connect to the Storage, get the list of containers and the ASPX then processes the container list, populates the form and delivers it back to the client. Given the low traffic usage, this didn’t exactly tax the server in any way, but there’s a much better way to handle this, and that’s by applying MOOP programming principles.

The first thing we need to do is to separate out the components of the blob storage interface. There are three parts:

The user interface
The interaction with Azure Blob Storage
The processing of results from Azure Blob Storage

The first step was to expose the functionality of the AzureCommands in a way that would make it easy for JavaScript to access. To do this, I created a Handler. Handlers seem scary but a Handler is really just a specialized bit of code with no UI that handles web based requests. You can find the Azure Blob Storage handler code here (I’ve actually added a project in AzureCommands to store the Blob, Table and Queue handler). All the handler does is grab parameters out of the request, make the appropriate call to the AzureCommands interface and then format the return data. This means that we can now interact with the Azure Storage elements from a web page without having all of the code that went into the ASPX.

Now that we have that piece of the puzzle, we can use JavaScript to write both our UI and to process the results of the interaction. Which means that a majority of the processing have been removed from our web server and shifted it to the client. By cutting the amount of processing that our web server has to do from interfacing with Azure Storage and processing the results to just interfacing with Azure Storage and returning the results. The fact that our web server is doing less means it can serve more requests, which means we can handle more traffic before we need to implement an additional web server. The only other thing our server needs to do is to deliver up the static HTML/Javascript files. And, if caching is properly implemented (as I discussed here), we can cut down on the number of files we need to serve up.

Tomorrow, I’ll discuss how we can capitalize on Azure Blob Storage to implement this caching correctly and discussing Handlers in much more detail as we look at the first of the Azure MOOP Framework handlers.

Technorati Tags: Windows Azure,MOOP Programming Principles,Azure MOOP Framework,Azure Architect

Caching is Good, HTTP Caching is Better

azuredba@gmail.com — Sat, 13 Nov 2010 15:25:51 GMT

Caching is a great way to cut down on processing. A client can say, “Hey, I have this version of an object, is that the same one you have?” And the server can either say, “Yep, they match” or “No, here, have the latest and greatest version.” Ok, that’s a simplified version but the reality is just as good. Take an HTML web page that has several JavaScript files and CSS files and a couple of images, not to mention the HTML that makes up the page. That could easily add up to over a MB. The table below is the base files and that’s not counting any of the image files or nested files the CSS may come up with.

Object	Size
dhtmlx.css	129 KB
BaseLibrary.js	5 KB
dhtmlx.js	1,039 KB
CipherSolverBanner.gif	1 KB
HTML in page	30 KB
Total	1,204 KB

Well, actually, the client passes in the “If-None-Match” header with an ETag and the server compares that ETag with the one it has for the local version and can respond with an HTTPStatusCode of NotModified (304) or with a full blown version of the object. Not only that, a number of the CSS and JavaScript files may be reused on multiple pages. And we pay $0.15 / GB, which may not seem like a lot, but let’s take a look at a simple scenario. You have a dynamic html page that uses AJAX and the size of the physical files is 1 MB. In addition, each users generates an additional Megabyte of Ajax created calls. That means that every thousand users/day will cost you $0.30 in bandwidth costs, half in Ajax created calls and half in static files. But it doesn’t have to cost that much. If you can enable caching you can drastically cut down on those costs.

Let’s say, just to begin with, that you can deliver all of your static files in a way that each individual user can cache them. That means that the first time each of those thousand users hits your web site, they are going to download the megabyte of static files plus they will generate the megabyte of Ajax calls so your cost / thousand users day one will be $0.30 but the cost on subsequent days will be $0.15. Over the course of a month that changes things from $9.00 ($0.15/day in static files + $0.15/day in Ajax calls over 30 days) to $4.65 (one time cost of $0.15 in static files and the $0.15day in Ajax calls over 30 days).

But wait, it gets better. Because there exist a number of intermediary cache servers between your clients and your server that pass information between your clients’ PCs and your server. If you happen to be able to specify the Cache Control headers to set it to be Public then all of those servers between your clients and your server are allowed to cache it. Which means the following is a very plausible scenario:

User A opens your URI in the browser
The HTTP request is routed through cache server XYZ which checks to see if it has a copy of the html. It doesn’t so it passes the request along.
Your server gets the request and returns the HTML with an ETag and Cache-Control: public and an Expires of one hour from now
Server XYZ keeps a copy of the HTML, ETag and Caching information and passes it back to User A.
User A’s browser interprets the markup and ends up requesting more information from your server, passing through cache server XYZ and the process is repeated.
User A now makes an AJAX request which is not cacheable.
The HTTP request is routed through cache server XYZ which checks to see if it has a copy of the request. It doesn’t so it passes the request along.
Your server gets the request and returns the Ajax information along with Cache-Control: nocache.
Server XYZ gets the Ajax information, notes the nocache and passes it back to User A without keeping a copy of it.

Now we have a cached version of the HTML (and images and CSS, etc) in sitting in cache server XYZ. This means that:

User B opens your URI in the browser
The HTTP request is routed through cache server XYZ which checks to see if it has a copy of the HTML. It does and that HTML hasn’t expired so it returns that HTML to User B.
This is repeated for all of the content you marked as cacheable.
User B now makes an Ajax request.
The HTTP request is routed through cache server XYZ which checks to see if it has a copy of the request. It doesn’t so it passes the request along.
Your server gets the request and returns the Ajax information along with Cache-Control: nocache.
Server XYZ gets the Ajax information, notes the nocache and passes it back to User A without keeping a copy of it.

This means your server never sees the request for the HTML from User B so you don’t end up paying bandwidth charges for it. Which means your monthly cost per thousand users will be closer to $4.50 than $4.65. But it also means that your server is free to handle the interesting things (Ajax responses, for example) instead of having to deliver up static content.

This is, of course, an over simplification of the process but one that illustrates the general principle. Unfortunately, it’s not something that’s implemented everywhere. But I’ve just made a modification to AzureCommands to add CheckBlobCache which check to see if an ETag matches in order to enable returning a 304 (Not Modified). I’ll be covering this in far more detail in the upcoming week as I reveal and explain the Azure MOOP Framework.

Technorati Tags: Azure Framework,HTTPCache,Cache,MOOP Programming Principles

Building Scalable Websites With Microsoft Azure and MOOP Programming Principles

azuredba@gmail.com — Fri, 12 Nov 2010 20:40:35 GMT

Some people might look at the title of this blog post and wonder what I’m talking about. “The whole point of Azure,” they might say, “is scalability!” Which is true but only in a way. Azure Web and Worker Roles are scalable in the fact that you can turn on as many as you’d like to meet demand. This is very useful if your website is featured on the national news or Slashdot or Fark, but it can be costly too. The goal of a website should be to serve as many pages as possible before you need to add that next instance. And that’s what I am talking about with Building Scalable Websites, going somewhat outside of Microsoft’s standard playbook for building websites and moving into a way to build a website that’s as efficient in delivering content as possible. I am a small company, I don’t have a budget that can afford any more compute hours than absolutely necessary. Fortunately, building a scalable website tends to minimize those compute hours.

There are a lot of paradigms that programming seems to shift through. I’m not the first to compare web sites to mainframes, but I think that today’s web has moved past the mainframe/dumb terminal phase of life and into the mainframe/pc phase of life. When the web was an infant, web servers delivered up static content. Then they began running scripts. When Microsoft entered the fray with IIS and Active Server Pages (ASP), they were scripts I knew how to write. This ASP had a lot of functionality in it and every keystroke might generate a return to the web server to process that script. If clicking a checkbox should make some text appear, the click would be sent to the server which would process the new page and re-render it back to the browser. This was the same type of behavior that dumb terminals used with main frames.

With time, however, came progress. ASP gave way to compiled ASPX, which cut down on some of the server processing but there were still a lot of round trips because the client, the web browser, was still pretty simple. Granted, you could write an executable program for your users to download and install, but this generated it’s own unique headaches, not the least of which was making sure users had the latest version of the software installed.

Today we live in a world where browser are far more intelligent then they used to be. Javascript and CSS can do miraculous things, HTML 5 (check out this website to see what it can do) promises even more. And yet, we still write code that is server-centric. In many ways, programmers like myself approach writing server side code as though we are programming a mainframe used solely by dumb terminals instead of terminals where we can offload processing that’s not essential to clients who have the intelligence to handle it. It does take a little more planning and forethought, but the benefits are worth it, because the less code that needs to run on the server, the more clients it can handle before we need to add instances.

I’m basing this framework on MOOP programming principles. MOOP stands for Matter Out Of Place, often used to talk about littering but it can also talk about clutter or anything that isn’t where it should be. And, while I’m developing a MOOP framework for Windows Azure, the MOOP Programming principles can be applied to almost any type of programming.

So, what are the MOOP programming principles? A place for everything and everything in its place. Which means defining the various places you have to work with, defining what determines that an object lives in a particular place and defining how all of the places fit together.

Within Azure, I can see the following places:

SQL Azure
Web Role
Worker Role
Blob Storage
Client browser

In addition, these places can be further broken down. The Web Role might have very specific, one use objects and it might have very generic use objects defined as well. One example of this is that my framework uses a generic handler to handle 99% of the interactions with the database. But some interactions require a specialized routine that the generic handler can’t handle as well for various reasons. In those instances, it is necessary to deploy an actual handler specifically for that one task.

Over the next week, as I roll out blog entries related to both MOOP programming principles and the Azure MOOP Framework, this will all become a lot clearer. But this post is really just intended to start you thinking about where your programming efforts are going and to ask you if your web server is working like a mainframe with dumb clients or one with smart clients that can take some of the processing and offload it from your webserver?

Until next week,

Josef Finsel,
Azure Evangelist and Sr. Consultant, Azure-Architect.com

Technorati Tags: Azure Framework,Programming Principles,Patterns,MOOP Programming Principles

Framework Posts Coming but First

azuredba@gmail.com — Wed, 10 Nov 2010 14:20:25 GMT

I’m wrapping up work on the first version of my Azure Framework, hoping to get it documented and posted soon, but in doing my work I’ve made a minor change to the AzureBlobLoader utility, a command line utility that loads files to Azure Blob storage. It now supports the Archive flag, meaning it will only upload files in a directory that have changed, cutting down on processing time and costs.

More...

Windows Azure Authentication Best Practices

azuredba@gmail.com — Mon, 18 Oct 2010 14:30:42 GMT

Microsoft’s Windows Azure has had a pretty stable authentication scheme since September 2009. I spent time this weekend updating the AzureCommands library to implement the latest changes to Azure. The biggest headache I encountered was implementing the new Authentication scheme. It turns out there were a couple of things MS did that I didn’t expect and a bug in my own code that caused some of my grief. Here I’ll document some of what I had to do to implement the new scheme and finish with a Best Practices Guidelines.

Defining the Changes

Microsoft made a number of changes, mostly in the canonicalized resource and canonicalized headers. These changes add more items to what makes up the string to generate the security hash, with more items supposedly being more secure. The new string includes the following values, separated by a new line (\n):

VERB
Content-Encoding
Content-Language
Content-Length
Content-MD5
Content-Type
Date
If-Modified-Since
If-Match
If-None-Match
If-Unmodified-Since
Range
Canonicalized Headers (different from earlier versions)
Canonicalized Resource (different from earlier versions)

The bold items are new but, as the documentation points out, “where there is no header value, the new line character only is specified.” This means a majority of the hash when creating non-web based interfaces tends to be just new line characters. I started out by modifying my CreateSharedKeyAuth stored procedure to include the new header values. Then it was time to tackle the Canonicalized Resource Changes.

The most obvious change is that the resource is not new line delimited, but there are a couple of other things. First, Query String parameters need to be sorted and can only appear once and their values must be sorted alphabetically.. Thus a legitimate URL that contains include=snapshots&include=metadata&include=uncommittedblobs would need to be include:metadata, snapshots, uncommittedblobs. So I created a new function to handle that and then created a CanonicalizeUrl2 function to implement all of the new rules.

And promptly got frustrated. Some things like list containers worked fine. Doing anything to container wasn’t working. This was because I had learned, through trial and error, that I needed to use just a new line character to get past the 401 Forbidden. Except that’s not exactly the case. And this was where I learned the first important lesson of the weekend, perhaps the most important: Fiddler is your friend.

When the server returns a 401 Forbidden error, it also returns a header showing the unhashed string it used to create the authentication. This means you can place the server’s string and your string side by side and figure out what’s wrong. Which lead me to realize that Content-Length should be an empty string when it’s 0 or a GET. Which is really HTTP 101 but it’s been awhile since I’ve done HTTP programming. Once I got Fiddler installed and running and worked through my AuthKey function so I would match, I ran into my second problem, which turned out to be minor modifications in the signature of some of the functions. So I spent some time working with the API docs to make the minor changes like new headers or new query string parameters. And then I ran into the biggest time sink and it turned out to be my own problem.

Metadata Naming Changes

The first problem I ran into was because I had code to automatically add metadata to show that data was added using my utility, some variation on “x-ms-meta-created-by: Finsel Azure Commands”. Unfortunately, “created-by” is no longer a valid name for metadata. Basically, if you can’t use it as a C parameter, you can’t use it as a metadata name. Once I finally tracked my 400 Bad Request errors to the metadata and changed it, things got easier. But it also exposed a problem.

If you created a blob using the pre-2009-09-19 authentication scheme, it could have metadata that will cause updating metdata using the new scheme to fail! Fortunately, I could blow away and reload my test data without consequences. However, it did get me thinking that I will probably write a utility this week that will walk through containers, blobs and queues and update the metadata names, probably to change anything after a – to be a capital letter (created-by becomes createdBy).

Utility Error

Once I fixed the MetaData naming problem, I ran into another issue of my own making. My AzureBlobLoader was appending an extra / into the blobs it loaded. This turned out to be an insidiously difficult error to track down because it threw 401 Forbidden errors. Once again, Fiddler came to the rescue, showing me the incorrectly formatted URLs and enabling me to figure out what the problem was and to fix it by correcting the utility, deleting the container and re-uploading it. I’ll actually talk more about this later this week when I discuss blobs and changes implemented in the 2009-09-19 version.

Table Storage

Table Storage turned out to only have one modification to the Lite authorization key, and that is the addition of two new headers, making it the easiest of all to implement.

Also, when they released the 2009-09-19 update, they started to return two new headers. x-ms-version is the version used to serve up the response. If you make a request using 2009-07-17 version you’ll see that value returned in the headers. Also, all of the services now return information about the server, “Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0” for example.

Best Practices

So, after working with and debugging the new authentication scheme, I think the following best practices are in order:

Always specify the version of authentication scheme you are using
Always use the same authentication scheme.
Technically, you can use the 2009-07-17 version for tables and the 2009-09-19 for blobs but once you’ve worked out the authentication scheme you can use it for everything and having the same authorization scheme simplifies debugging.
Always use the full authentication when possible.
While you can use a “lite” authentication for Blobs and Queues, it makes more sense to use the standard authentication. Again, it minimizes the amount of debugging and simplifies your programming.
Have Fiddler installed so you can see what differences you and the server are having.
Always implement the latest authentication scheme when possible, with full testing to see what impact those changes may have.

Coming up

Now that I have the latest and greatest changes implemented in AzureCommands, I’ll be laying out the ground work for some outside the box thinking that will help you implement Cloud Computing in some new ways.