IDrive .NET backup library

Posted on February 22, 2011 by Mikkel Ovesen

What does IDriveLibrary it do?

IDrive EVS gives you 5GB of free backup space on the IDrive platform. IDrive EVS makes it possible via a kind of web http interface to upload files and more.

The library I have written is called IDriveLibrary and can be found here https://bitbucket.org/ovesen/idrivelibrary. My thought about the project, was to be able to write a backup routine in C# for my servers.

Unfortunately IDrive EVS seems buggy and slow. I have in an earlier review of backup providers deemed IDrive buggy (http://blog.ovesens.net/2010/01/test-of-online-backup-providers/). Nothing has changed.

During the development of the library I have had to contact IDrive 5 times, either because documented features simply did not work or because of errors in the documentation.

Share|
Posted in Development, Software | Tagged , , , | Leave a comment

NHibernate Membase caching provider

Posted on February 22, 2011 by Mikkel Ovesen

Enyim has made a nice client library for both Memcache and Membase. Membase can be used as a free distributed caching platform.

With the new provider model introduced in ASP.NET 4, it is now possible to write providers for output caching and session state. Enyim has done just that with their memcache-provider.

I use NHibernate for some of my projects and being able to use a Membase cache provider for NHibernates second level cache would be very nice.

With help from Christian Dalager and my self, such a library is now available. Take a look here https://bitbucket.org/ovesen/membasecacheprovider

Zipped binaries are provided with mapping against NHibernate 2.0.1, 2.1.0, 2.1.2 and NH3 here https://bitbucket.org/ovesen/membasecacheprovider/downloads

An example configuration file is provided in the downloads, but else you can see the options here:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <!-- This is a sample configuration file for using MembaseCacheProvider -->

    <configSections>

        <!-- (Required) Default configuration section for Membase -->
        <section name="membase" type="Membase.Configuration.MembaseClientSection, Membase" />

        <!-- (Optional) An additional configuration section for Membase, can be used to target another bucket or servers -->
        <section name="membaseNhibernate" type="Membase.Configuration.MembaseClientSection, Membase" />

        <!-- (Optional) A section for the Enyim Membase client, allows loggint to e.g. log4net -->
        <sectionGroup name="enyim.com">
            <section name="log" type="Enyim.Caching.Configuration.LoggerSection, Enyim.Caching" />
        </sectionGroup>

        <!-- (Optional, but recommended) A configuration section that allows specific settings to be set for different NHibernate caching regions -->
        <section name="membaseNhibernateCache" type="MembaseCacheProvider.MembaseNhConfiguration, MembaseCacheProvider" />

    </configSections>

    <!-- (Optional) Defines that log4net should be used for Membase client logging, remember to define log4net configuration -->
    <enyim.com>
        <log factory="Enyim.Caching.Log4NetFactory, Enyim.Caching.Log4NetAdapter" />
    </enyim.com>

    <!-- (Required) Maps to the configuration section named "membase", defines a single server url and the "default" bucket -->
    <!-- Other configuration settings are available, look in the Enyim documentation and examples -->
    <!-- here http://memcached.enyim.com/ or here https://github.com/enyim/memcached-providers -->
    <membase>
        <servers bucket="default">
            <add uri="http://localhost:8091/pools/default" />
        </servers>
    </membase>

    <!-- (Optional) Maps to the configuration section name "membaseNhibernate", defines a single server url and the bucket "default_nhibernate" -->
    <membaseNhibernate>
        <servers bucket="default_nhibernate">
            <add uri="http://localhost:8091/pools/default" />
        </servers>

        <!-- (Optional) Sets specific settings for the socket pool -->
        <socketPool minPoolSize="10" maxPoolSize="100" connectionTimeout="00:00:10" />

        <!-- (Optional) Sets a specific locator -->
        <locator type="Enyim.Caching.Memcached.DefaultNodeLocator, Enyim.Caching" />
    </membaseNhibernate>

    <!-- (Optional, but recommended) Configuration settings of NHibernate second level cache, Membase provider -->
    <!-- The section attribute defines that the Membase configuration names "membaseNhibernate" should be used -->
    <membaseNhibernateCache section="membaseNhibernate">

        <!-- Here expiration details are specified for the Nhibernate cache regions LongTerm and ShortTerm -->
        <!-- The expiration are defined in minutes, meaning LongTerm = 60 minutes and ShortTerm = 2 minutes -->
        <cache region="LongTerm" expiration="60" />
        <cache region="ShortTerm" expiration="2" />
    </membaseNhibernateCache>

</configuration>
Share|
Posted in Development | Tagged , , , , | 18 Comments

Membase local development machine ip problem – FIXED

Posted on February 22, 2011 by Mikkel Ovesen

Membase caching stops working

I struggled with Membase last night. I added NHibernate 2. level caching, Asp.net Session caching, Output caching and custom caching to some new Membase providers.

It worked at first, but then suddenly nothing was cached, and nothing was retrieved from the cache. I was working on my development machine and it seems like the problem is due to the network switching.

Well here is how it was solved. First navigate to the Membase server directory (C:\Program Files\Membase\Server\bin). Then execute the following commands.

service_stop.bat
service_unregister.bat
service_register.bat ns_1@127.0.0.1
service_start.bat

Read the original here: http://blog.danhulton.com/2011/02/05/membase-on-windows-7-ip-address-fix/

Share|
Posted in Development, Software | Tagged , , , , , | Leave a comment

Membase cache item expiration issue

Posted on February 21, 2011 by Mikkel Ovesen

Membase can be used as a distributed caching platform, and best of all… it is free to use. This is from their website:

Unlimited use in development and up to two nodes may be deployed for free in a production cluster, with modest additional prices for additional nodes.

When I started to develope and test Membase… I first wondered. Why does the total number of caching items continue to increase. Why do the number never decrease?

I expected items to be removed when their expiration time passed, but they didn’t and I started to wonder whether I was doing something wrong.

It turns out that Membase does not have an active expiration cache clean-up functionality. If an item has expired, it is not removed until the next Get, or when the cache is full and items need to be evicted.

Here is a very good description of the logic (Original here):

Your understanding of evictions is correct. When a new item is added and there is no space for it, an older item must be thrown away. One key thing to understand is that expiration and eviction have nothing to do with one another.

When an item expires, nothing actually happens. It is only upon the next access of that item that the server will notice it has expired and remove it from memory. There is also no guarantee that already expired items will be evicted first, though it practicality they are usually old enough that they tend to be among the first to get evicted (but nothing in the server controls this).

Now, onto the nuances of evictions. Within memcached there is a slab allocator which handles memory allocation for items. Basically, the whole memory space is broken up into 1mb pages which are then broken up into slabs of varying sizes. There are configuration options to control this, but that’s the default mechanism. Each size is called a “slab class” and has its own eviction logic which means that if one slab class fills up, you can’t reclaim memory from other slab classes. Depending on the variation in size of the objects that you are putting in, this could become more or less of a problem for you. Can you telnet to your servers and run ‘stats’ and ‘stats slabs’ and paste the output? That will let me understand exactly where memory is being taken up in your server.

Another key point to understand is that evictions are not always a bad thing. With changing workloads you can end up with lots of stale data still stored in cache that just needs to take some time to be pushed out. If everything seems to be running fine (with the exception of the evictions) then you probably don’t really need to take any action. Are you seeing a large number of misses as well? High misses and high evictions can usually be correlated to mean that the application is expecting data to be there when it’s not, and that would be a problem. If the application is receiving the data that it’s asking for, your cache is working correctly.

Your understanding of evictions is correct. When a new item is added and there is no space for it, an older item must be thrown away. One key thing to understand is that expiration and eviction have nothing to do with one another.When an item expires, nothing actually happens. It is only upon the next access of that item that the server will notice it has expired and remove it from memory. There is also no guarantee that already expired items will be evicted first, though it practicality they are usually old enough that they tend to be among the first to get evicted (but nothing in the server controls this).Now, onto the nuances of evictions. Within memcached there is a slab allocator which handles memory allocation for items. Basically, the whole memory space is broken up into 1mb pages which are then broken up into slabs of varying sizes. There are configuration options to control this, but that’s the default mechanism. Each size is called a “slab class” and has its own eviction logic which means that if one slab class fills up, you can’t reclaim memory from other slab classes. Depending on the variation in size of the objects that you are putting in, this could become more or less of a problem for you. Can you telnet to your servers and run ‘stats’ and ‘stats slabs’ and paste the output? That will let me understand exactly where memory is being taken up in your server.

Another key point to understand is that evictions are not always a bad thing. With changing workloads you can end up with lots of stale data still stored in cache that just needs to take some time to be pushed out. If everything seems to be running fine (with the exception of the evictions) then you probably don’t really need to take any action. Are you seeing a large number of misses as well? High misses and high evictions can usually be correlated to mean that the application is expecting data to be there when it’s not, and that would be a problem. If the application is receiving the data that it’s asking for, your cache is working correctly.

So if it seems like the total number of your Membase  cache items just increases and increases, don’t worry it is actually normal behavior. Just make sure you have configured the server instance with the recommended amount of  server memory and disk resources. Take a look in the best practises.

Share|
Posted in Development, Software | Tagged , , , | Leave a comment

Sample TwitterTest authentication and console application

Posted on February 21, 2011 by Mikkel Ovesen

How to make your application or service tweet

My case: I have a windows service and I want to be able to know the internal state of this service and e.g. when an exception was thrown. The obviusl answer for this is logging. But I already use Log4Net, and I want to be able to access this info without having to remote to the server.
Others have solved this issue by making e.g. Kayak part of the service and expose that state as JSON. That do sound tempting, however I am more to some kind of logging mechanism. I know that there exists Twitter Appenders for Log4Net, but by using something existing I will not learn anything :)
So what I want to implement is custom Twitter logging, and here is how.

  1. Register your application with Twitter (http://dev.twitter.com/apps/new)
  2. Save your consumerKey and consumerSecret
  3. Run TwitterAuthenticator to retrieve the Access token. Save Token and TokenSecret
  4. Use the TweetConsole as a sample for Tweeting. Use the consumerKey, consumerSecret and the Access Token details

That is it…

You can find the source code here: https://bitbucket.org/ovesen/twittertest

Share|
Posted in Development, Software | Tagged , , | Leave a comment

Mercurial problems with Danish letters

Posted on February 1, 2011 by Mikkel Ovesen

Mercurial can have problems with handling non ASCII letters in file names, comments etc. when it across different platforms (Windows, Unix, MacOSX). I have even experienced the problem within different versions of Windows.

The problem: File names containg e.g. æøå is renamed to æøå…

This solution worked for me. I use TortoiseHG which is installed in c:\program files (x86)\TortoiseHG.

  1. Create a folder called extensions in c:\program files (x86)\TortoiseHG
  2. Extract the content of the following file to the extensions folder (hg-fixutf8 or access the files here https://bitbucket.org/stefanrusek/hg-fixutf8)
  3. You should now have a path to the following file: c:\Program Files (x86)\TortoiseHg\extensions\hg-fixutf8\fixutf8.py
  4. Open C:\Users\[Username]\mercurial.ini in Notepad or another text editor
  5. Add the following extension so your mercurial.ini looks something like this
# Generated by TortoiseHg setting dialog

[ui]
username = [YOUR USERNAME]

[extensions]
fixutf8 = C:\Program Files (x86)\TortoiseHg\extensions\hg-fixutf8\fixutf8.py

With that in place, the first time you update your working copy file names will be corrected. If you already have updated your working copy, then do a clean update.

Share|
Posted in Development, Software | Tagged , , | Leave a comment

Umbraco TinyMCE converting absolute to relative url

Posted on January 31, 2011 by Mikkel Ovesen

Well here is how you work around the issue.

Open the config/tinyMceConfig.config and add the lines 5 and 6 to your config:

<!-- this area is for custom config settings that should be added during TinyMCE initialization -->
<customConfig>
<config key="entity_encoding">raw</config>
<config key="spellchecker_rpc_url">GoogleSpellChecker.ashx</config>
<config key="relative_urls">false</config>
<config key="convert_urls">false</config>
</customConfig>

Please note that this only fixes the issue partly. It seems like Umbraco does a convertion on save. My solution was to link to the url without www.

Share|
Posted in Development | Tagged , | Leave a comment

Async Httphandlers in ASP.NET

Posted on May 27, 2010 by Mikkel Ovesen

What is the correct way to create a httphandler that potentially can take time and resources to complete.

Well this article explains it very well:
http://msdn.microsoft.com/en-us/magazine/cc164128.aspx

If you don’t want to read all, you find the best solution in the bottom of the article.

But you can also find the code here, download this library C# Threadpool.

AsyncRequest

class AsyncRequest
{
  private AsyncRequestState _asyncRequestState;

  public AsyncRequest(AsyncRequestState ars)
  {
    _asyncRequestState = ars;
  }

  public void ProcessRequest()
  {
    // This is where the non-cpu-bound activity would take place, such as
    // accessing a Web Service, polling a slow piece of hardware, or
    // performing a lengthy database operation. I put the thread to sleep
    // for two seconds to simulate a lengthy operation.
    Thread.Sleep(2000);

    _asyncRequestState._ctx.Response.Output.Write(
            "AsyncThread, {0}",
            AppDomain.GetCurrentThreadId());

    // tell asp.net I am finished processing this request
    _asyncRequestState.CompleteRequest();
  }
}

AsyncHandler

namespace EssentialAspDotNet.HttpPipeline
{
  // AsyncRequestState and AsyncRequest remain the same
  // as in the previous example

  public class AsyncHandler : IHttpAsyncHandler
  {
    static DevelopMentor.ThreadPool _threadPool;

    static AsyncHandler()
    {
      _threadPool =
        new DevelopMentor.ThreadPool(2, 25, "AsyncPool");
      _threadPool.PropogateCallContext = true;
      _threadPool.PropogateThreadPrincipal = true;
      _threadPool.PropogateHttpContext = true;
      _threadPool.Start();
    }

    public void ProcessRequest(HttpContext ctx)
    {
     // not used
    }

    public bool IsReusable
    {
      get { return false;}
    }

    public IAsyncResult BeginProcessRequest(HttpContext ctx,
                     AsyncCallback cb, object obj)
    {
      AsyncRequestState reqState =
                     new AsyncRequestState(ctx, cb, obj);
      _threadPool.PostRequest(
                     new DevelopMentor.WorkRequestDelegate(ProcessRequest),
                     reqState);

      return reqState;
    }

    public void EndProcessRequest(IAsyncResult ar)
    {
    }

    void ProcessRequest(object state, DateTime requestTime)
    {
      AsyncRequestState reqState = state as AsyncRequestState;

      // Take some time to do it
      Thread.Sleep(2000);

      reqState._ctx.Response.Output.Write(
                   "AsyncThreadPool, {0}",
                    AppDomain.GetCurrentThreadId);

      // tell asp.net you are finished processing this request
      reqState.CompleteRequest();
    }

  }
}

AsyncPage

namespace EssentialAspDotNet.HttpPipeline
{
 public class AsyncPage : Page, IHttpAsyncHandler
  {
    static protected DevelopMentor.ThreadPool _threadPool;

    static AsyncPage()
    {
      _threadPool =
       new DevelopMentor.ThreadPool(2, 25, "AsyncPool");
      _threadPool.PropogateCallContext = true;
      _threadPool.PropogateThreadPrincipal = true;
      _threadPool.PropogateHttpContext = true;
      _threadPool.Start();
    }

    public new void ProcessRequest(HttpContext ctx)
    {
      // not used
    }

    public new bool IsReusable
    {
      get { return false;}
    }

    public IAsyncResult BeginProcessRequest(HttpContext ctx,
                                            AsyncCallback cb, object obj)
    {
      AsyncRequestState reqState =
             new AsyncRequestState(ctx, cb, obj);
      _threadPool.PostRequest(
             new DevelopMentor.WorkRequestDelegate(ProcessRequest),
             reqState);

      return reqState;
    }

    public void EndProcessRequest(IAsyncResult ar)
    {
    }

    void ProcessRequest(object state, DateTime requestTime)
    {
      AsyncRequestState reqState = state as AsyncRequestState;

      // Synchronously call base class Page.ProcessRequest
      // as you are now on a thread pool thread.
      base.ProcessRequest(reqState._ctx);

      // Once complete, call CompleteRequest to finish
      reqState.CompleteRequest();
    }
  }
}
Share|
Posted in Development | Tagged , , , , | Leave a comment

Manually start Visual Studio dev. web server

Posted on March 19, 2010 by Mikkel Ovesen

I needed to manually start the Visual Studio dev. web server for profililng purposes…

The web server is located here:
C:\Program Files (x86)\Common Files\microsoft shared\DevServer\9.0\WebDev.WebServer.EXE

Arguments

/port:<port number> Optional, an unused port number between 1 and 65535.
/path:<physical path> A valid directory where the web application is rooted.
/vpath:<virtual path> Optional, the virtual path or application root in the form of ‘/<app name>’.

Example

C:\Program Files (x86)\Common Files\microsoft shared\DevServer\9.0>
WebDev.WebServer.EXE /port:8080 /path:"C:\Users\Administrator\Documents\Visual Studio 2008\Projects\WebApplication\Compiled" \vpath:"\"
Share|
Posted in Development, Software | Tagged , , | Leave a comment

Google Sitemap .NET links

Posted on March 2, 2010 by Mikkel Ovesen

Some links with information about creating a Google Sitemap either directly in XML or via some .NET code.

http://gerardmcgarry.com/blog/creating-a-google-sitemap-aspnet-website

http://www.codeproject.com/KB/aspnet/GoogleSiteMapProvider.aspx

Share|
Posted in Development, Personal | Tagged , , , , | Leave a comment