SPFolder.WelcomePage throwing access denied

I was developing a web part for a client which, given a URL copy/pasted from another source, had to resolve to the appropriate list or library item to display some information about it. While these URLs were obtained from another system, they were pretty well guaranteed to map to an item in SharePoint. Mapping the URLs is easy enough using either SPWeb.GetListItem (depending on the URL format), but it was also possible I'd be passed the URL of the web itself, which I would need to resolve to the welcome page library item. For example, given


I would have to resolve to


Or whatever the home page was. Not a big deal right? Just use SPWeb.GetFile. Worked perfectly. Yay, blog post over!

Then it went to test.

A few users were reporting they'd sometimes get Access Denied errors, yet when they browsed to the URL they could see it fine. Getting some examples, they were all URLs of the SPWeb rather than a URL of an exact list/library item.

After a bit of digging around, I'd found out these users were in a particular SharePoint group, and this group had a custom permission level assigned. The only difference this custom permission level had, was that "Browse Directories" had been removed. Sure enough, lowering my account on my dev setup to this permission level replicated the problem, and I was able to determine that the call to UnauthorizedAccessException.

I could use SPSecurity.RunWithElevatedPrivileges to access that property, but it seems like an unnecessary hack. Besides, how does SharePoint redirect users to the welcome page when they browse to the web URL? I know SharePoints inner workings are magical, but surely this property is involved somewhere.

Time to fire up ILSpy (like I need an excuse to do that). The first thing I did was look at the implementation of SPWeb.RootFolder.WelcomePage:

// Microsoft.SharePoint.SPFolder
public string WelcomePage {
    get {
        // ok not exactly, but this is the nett result
        return (string)this.Properties["vti_welcomepage"];

Not too exciting, but it did reveal why the error was happening: accessing SPFolder.Properties apparently requires Browse Directories permissions. Analysing its usage was a bit more interesting though. It's used in a bunch of places, but one caught my eye:

Microsoft.SharePoint.Utilities.SPMobileUtility.GetWelcomePageUrl(SPWeb web)

Opening this method I was hoping to be dazzled, instead…

// Microsoft.SharePoint.Utilities.SPMobileUtility
internal static string GetWelcomePageUrl(SPWeb web)
    if (web.DoesUserHavePermissions(SPBasePermissions.BrowseDirectories))
        return web.RootFolder.WelcomePage;
    string welcomePage = string.Empty;
    SPSecurity.RunWithElevatedPrivileges(delegate {
        using (SPSite sPSite = new SPSite(web.Site.ID))
        using (SPWeb sPWeb = sPSite.OpenWeb(web.ID))
            welcomePage = sPWeb.RootFolder.WelcomePage;
    return welcomePage;
Face Palm

At least I was on the "right" track using SPSecurity.RunWithElevatedPrivileges

I'm a contributor!

Over the past few years I've been primarily developing on the SharePoint platform, which typically means I'm working on an intranet. So for the most part, everything I've worked on has really only used by people who are already pissed off enough at having to use SharePoint, let alone some overpaid consultants solution that doesn't cater to everyone's personal requirements. No one really sees my work… I can't remember the last time I had a code review (in fact, I don't think I ever have, as a consultant or otherwise) – it generally just works, but I have no idea whether my code is good.

When I stumbled across Prism and realised it didn't have ASP.NET syntax highlighting, rather than throwing my hands up cursing the lack of what I needed, I decided to get my hands dirty. Finally my GitHub account was going to be used! After getting my head around git again (SourceTree to the rescue, damn I've missed git), I forked the project, built an extension to highlight ASP.NET, and submitted a pull request… which still hasn't been accepted =(

But, I have submitted a few bug fixes/enhancements since which have been merged, so yay! When I got the email my first PR had been merged, I was in the middle of shovelling lunch in my face and almost choked. Something I'd written is now live on a site that on its first day live had 23,000 unique visitors. That's scary and cool at the same time =) Another fix I'd submitted, three minutes after submitting the PR I'd received that it had been merged:

That is awesome!

But this also made me feel so out of date. "Developing" on SharePoint is a term used fairly loosely. Most clients don't want customisation (especially not code customisation), and even those that do typically need dribs and drabs of "plugin" code; events handlers or ribbon controls. It's not overly creative or challenging, and you tend to spend more time trying to figure out the intricacies of the platform to fit your square-peg code through the round-hole framework.

Then I see people all around the world collaborating on these great projects flooding through at least weekly on G+ or Twitter, and I can understand why some people say programming with .net is like cooking in a McDonald's kitchen. I feel rusty. This industry already moves fast as it is, I feel like I've missed out on so much by working within a platform/framework language.

On the other hand, perhaps having such a flood of information from various social networks just makes you feel more inadequate than you really are. It shouldn't be any surprise that following some of the top people in the world will come up with a steady stream of cool stuff. More so, there are lots of people working on lots of different things, why am I comparing my self to the output of dozens of the worlds best? There will always be someone better than me. Perhaps I should just be happy I'm now a part of the 9%* of people who actually contribute, instead of being disappointed I'm not in the 1% (yet). At least now I have a sliver of code out there for the world to see. After all:

It's not who you are underneath, it's what you do that defines you

Yes, I just quoted from a Batman movie. No I'm not proud of it, but that one line struck a chord with me!

* I'd love to know the actual stats for this on GitHub

SharePoint DateTimeControl and Labels

I have been doing a bit of accessibility work for a clients public facing website recently. One of the key areas was ensuring the feedback form was accessible to users with screen readers, which meant ensuring inputs have appropriately associated labels (amongst other things).

For most standard inputs this isn't a problem - text boxes, radio/check box lists work well enough with the ASP.NET Label controls. However, they were also using the SharePoint AssociatedControlID attribute won't have the desired effect, since at render time the "control" will just be a div. Consequently, when a vision impaired user focuses on the date input field, they don't get any hints read out. Furthermore, clicking a label will normally move focus to the associated text box input; but this functionality is also lost.

As part of my attempts to validate the DateTimeControl, I came across ControlToValidate property to {ControlId}${ControlId}Date and it will validate the actual date part of the datetime. Neat!

So I tried this with the AssociatedControlID property of the Label control:

<asp:Label runat="server" ID="FaultDateLabel" AssociatedControlID="FaultDate$FaultDateDate" Text="Fault Date" />

It worked beautifully!

  1. When tabbing to the date portion of the DateTimeControl while using a screen reader, the software correctly read out "Fault Date"
  2. When clicking the "Fault Date" label, focus was given to the date portion of the DateTimeControl.

Awesome :)

But one thing annoys me about these "static" Label controls… why do they need to be a server control? I'm not accessing it via code, nor referencing it in any other why, so why add another control to the control tree (and probably more to the ViewState)? I tried the following, and was pleasantly surprised:

<label for="<%= DateFrom.ClientID %>__DateFromDate">Fault Date</label>

It worked just as well. A couple of things to note:

  1. You need to use ClientID, not ControlID, since we need to output the long horrible generated ID (when using auto IDs)
  2. The generated ID delimits each nested control with a double underscore, not a dollar sign like when referencing the nested control server side

I wasn't sure whether this is best practice or if it has any performance implications, but after a quick read of Improving ASP.NET Performance, it seems this isn't so bad (and actually recommended for heavy pages or controls) as it writes directly to the response stream.

Working around GetDisplayGroup throwing "Specified cast is not valid" via PowerShell

I ran in to an issue today trying to assign some scopes to one of the standard display groups (Search Dropdown) on a newly created site collection, and ran in to this little problem:

$group=$scopes.GetDisplayGroup($siteUrl,"Search Dropdown")
Error Calling GetDisplayGroup with 2 arguments, Specified cast is not valid

A quick Google yielded some examples but no solutions other than manually adding the groups, or visiting /_layouts/viewscope.aspx – which inexplicably fixed the problem.

As others had discovered, inspecting all display groups (or using GetDisplayGroupsForSite) confirmed no display groups present for that site, yet when viewing /_layouts/viewscopes.aspx the groups are present (and subsequently also available via code). So, is viewscopes.aspx actually creating the scopes if they don't exist?

It sure does! Kind of. Starting from InitializeGridView in the code behind for viewscopes.aspx, here are the relevant calls:

// Microsoft.Office.Server.Search.Internal.UI.ViewScopesPage.InitializeGridView(bool) (called from OnPreRender)

// Microsoft.Office.Server.Search.Administration.ScopesUtilities.EnsureConsumer(ScopesManager)
scopesManager.Consumers.EnsureRegistered(asConsumerName); // asConsumerName=SPSite.Id.ToString("D")

(Lots of boring calls to register the consumer, until we get to…)

// Microsoft.Office.Server.Search.Administration.Scopes.OnConsumerAdded(Consumer)
Scope scope = this.AllScopes [1]; // magic!
Uri url = consumer.Url;
ScopeDisplayGroup scopeDisplayGroup = this.AllDisplayGroups.UnprotectedCreate("Search Dropdown", "(description)", url, true, true);
if (scope != null)
  scopeDisplayGroup.UnprotectedAdd (scope);
  scopeDisplayGroup.UnprotectedSetDefault (scope);

(Note: calls are abbreviated a bit just for clarity, they pull in resource strings and do all sorts of other magic).

Great, loading viewscopes.aspx creates the 'Search Dropdown', 'Advanced Search' and 'Site Directories' display groups against the site collection (eventually). So all I have to do is call an appropriate method somewhere along the line and have the scopes created for me. Right? Not a chance. All methods are private or within internal classes, and the only entry points are from ASPX page code behinds. Yuck!

But all is not lost. Thanks to this page, I can actually invoke the private methods, and get these scopes created:

# Pull in our references
Add-PSSnapIn Microsoft.SharePoint.Powershell -ErrorAction:SilentlyContinue
[reflection.assembly]::LoadWithPartialName("Microsoft.Office.Server") | Out-Null

# Get our site, search context and scopes
$site = Get-SPSite $siteUrl
$siteId = $site.Id.ToString("D")
$ctx = [Microsoft.Office.Server.Search.Administration.SearchContext]::GetContext($site)
$scopes = New-Object Microsoft.Office.Server.Search.Administration.Scopes($ctx)

# Here's the cool stuff. We're really just calling $scopes.Consumers.EnsureRegistered($siteId)
$BindingFlags = [Reflection.BindingFlags] "NonPublic,Instance"

# $consumers = $scopes.Consumers
$consumers = $scopes.GetType().GetMember("Consumers",$BindingFlags)[0].GetValue($scopes, $null)
# $consumers.EnsureRegistered(…)
$consumer = $consumers.GetType().GetMethod("EnsureRegistered",$BindingFlags).Invoke($consumers, $siteId)

It works beautifully! Now, you may be thinking this is overkill. Why not just do one of the following:

  1. Use powershell hit /_layouts/viewscopes.aspx (which will handle all of this nonsense for us)
  2. Create the scopes yourself, instead of sorcery invoking private methods Here's why:

Hit viewscopes.aspx via PowerShell

Yes, this is a viable solution (and arguably a less hacky one). However, if this is a new site collection, there will probably be significant warm up time waiting for the w3wp processes and what not to fire up, so you may have to try hitting the URL a few times. It also annoys me that I have to make a dumb web request to get something done (and by "dumb" I mean blindly hitting a URL, not caring about input and disregarding the output). I'm developing against this platform, I should be able to do this via code!

Create the scopes myself via PowerShell

Seems fair. However the code in OnConsumerAdded unfortunately isn't expecting something else to be creating its scopes, and doesn't cater for it. When I said, OnConsumerAdded adds the scopes, it does just that. It doesn't check to see if they exist and create otherwise, it just attempts to create them, and an exception is thrown if they're already there. Which means the first time someone visits viewscopes.aspx, they're going to get an ugly error message (and potentially, not have the site collection registered as a consumer, and I don't even know what that means! :-) ). The assumption seems to be that no-one is going to try to programmatically use display groups until a human as actually accessed that page. No, that would never happen! /s

Furthermore, these scopes are SharePoint specific display groups. You shouldn't have to be dealing with that yourself. That's like having to manually create your Pages library because the publishing feature will only do it for you if it's a Tuesday. And you're wearing a pink shirt.

So for me, sneakily calling Scopes.Consumers.EnureRegistered() is the best solution. It won't break viewscopes.aspx and you're not replicating the work SharePoint already does (as much) to create display groups.

The last thing which bugged me was this: why is GetDisplayGroup throwing an illegal cast exception? Some people suggested permissions (not being able to access SPSite.Url), but I think if that were the case, you'd have much bigger problems anyway, you're trying to manipulate search display groups after all!

It's actually nothing to do with this method, or what you're passing to it. GetDisplayGroup makes a call to GetDisplayGroupIDFromName(string siteId, string name), which in turn calls stored proc dbo.proc_MSS_GetScopeDisplayGroupIDFromName to return the ID of the group name, like so:

result = (int)sqlCommand.Parameters ["@DisplayGroupID"].Value;

However, if the group doesn't exist (for instance, no one's hit viewscopes.aspx yet), the stored proc returns System.DBNull. And gosh darn it, trying to cast System.DBNull.Value to an int isn't taken too kindly 'round these parts. And that's where the invalid cast exception comes from.

So I guess the take away message here is this: don't use GetDisplayGroup unless you're damn certain the display group exists. If you're unsure, it might be better to use GetDisplayGroupForSite:

$groups = $scopes.GetDisplayGroupForSite($site.Url) 
$group=@($groups | ?{ $_.Name -eq "Search Dropdown" })[0]
if ( -not $group ) {
  // create it

How Dolphin Browser cost me $180

Most of us have probably heard stories of people travelling overseas with international roaming enabled, and coming home to massive bills afterwards. Mine isn't quite that interesting, but it did happen within my own country (no roaming) while I wasn't even using my phone.

The short story… Dolphin Browser somehow managed to download almost 2 gigabytes in about an hour while my phone was locked in my pocket.

The long story…

____I was just about to head out to dinner, and noticed my phone had 50% battery left. Great, that'll last me. A couple of hours later I was going to show someone a video I'd recorded on my phone. Pulled my phone out of my pocket… dead. Annoying, but my phone has shut itself down randomly before, so didn't think too much of it. I fired it up and just caught a glimpse of a red exclamation mark on the battery icon as it it immediately turned itself back off. Bugger.

I figured while I was sitting at dinner, the screen was being kept on by my leg activating the lock pattern through my pocket… not the first time it happened! It wasn't until I got back to my hotel room and plugged in power that I saw a data limit warning. "Surely not", I thought, "it's five days in to the month, I've only used about 300 MB!" So I opened up the data usage app, and saw this:

Android mobile data usage, showing 1.95 gigabytes used by Dolphin Browser
Dolphin data hog

How could this possibly be? I hadn't used my phone in all that time, and the last thing I had open in Dolphin was a news site. I don't think I could have downloaded that much from browsing if I tried, and yet Dolphin had done it while my phone wasn't even awake!

Android mobile data usage specifically for Dolphin Browser, showing 1.74 gigabytes used in the background
Pretty much all background data

So I immediately disabled 3G data, and tried to figure out how much I'd get stung; about 650 MB over quota, times 20cm/MB… About $130. Shit. And that's assuming the data counted here matched what my provider counted, I could only cross my fingers and hope it was less. Significantly less.

It wasn't. It was a few hundred megabytes more in fact, so I'm looking at about $180 excess usage charges. Had this actually been legitimate usage such as YouTube or streaming music it wouldn't be so bad, but to be out of pocket because of shitty programming is frustrating. Having said that, thankfully my phone was only 50% charged, more so that it wasn't on charge overnight. I can't imagine the damage that would have done!

Finally, yes I'm aware there is an inbuilt feature to disable mobile data once configurable limits are reached, and I normally have this enabled. However I didn't this month because last month I legitimately "exceeded" my quota according to android, but in actual fact I'd used a lot of free traffic. In any case, not having this feature enabled doesn't excuse Dolphin from hammering my connection in the background, and even if it were enabled, I'd still be 25 days away from being able to use my data connection again.

I can only hope my provider will take some pity on me, realise it wasn't legitimate usage and either credit the charge back or work something out to lessen the pain.

Finally, I'm somewhat impressed. According to my usage graphs from my provider, all data usage was within one hour:

Graph showing all data usage occurred within a one hour period
Absolutely hammered!

That's impressive! 1.74 gigabytes in less than an hour on 3G. Either that's a data counting issue, or that was sustained 500 kB/s on average over the hour!