Query SharePoint Document Library using CAML

SharePoint exposes multiple APIs so user can access the SharePoint list using the Collaborative Application Markup Language (CAML) query.

In this article, we will discuss how to use HttpClient in asp.net core to access the SharePoint document library.

Authentication API Endpoint

Before you can make any SharePoint API calls, you must first obtain a form digest from SharePoint. A form digest is a client-side token provided by SharePoint that authenticates a credential to perform operations on a specific SharePoint site or site collection. The form digest value includes a token and a timeout date and time, delimited by a comma (,) character.

The API endpoint URL to get the form digest is:

<YourSharePointSiteUrl>/_api/contextinfo

Make a POST request to this URL, you will get the JSON response that contains form digest value as following.

The value of the “FormDigestValue” property inside above json response is the value you need. Add the value to the request header called “X-RequestDigest” in every following API call.

List API Endpoint

The list search API endpoint allows you to search the lists in SharePoint site. The list API endpoint URL is:

<YourSharePointSiteUrl>/_api/web/lists

You will get JSON response that contains all the lists by sending a GET request to this URL. Following is the sample of one list information:

SharePoint API query follows OData spec. You can use $filter query string parameter to filter out the lists you want. You can also use the $select query string parameter to retrieve only the information inside a list you need. You can get more information about OData Uri conventions from here: https://www.odata.org/documentation/odata-version-2-0/uri-conventions/

In order to filter out only the Document Library (BaseType is 1 and BaseTemplate is 101), we append following query string to the API endpoint:

$filter=Hidden+eq+false%20and%20BaseType+eq+1%20and%20BaseTemplate+eq+101

If we only want to get the BaseType, BaseTemplate and Title information of each list, we append following $select query string to the end:

&$select=BaseType,BaseTemplate,Title

Document Search API Endpoint

Once you have the document library title, you can search the items inside the document library using following API endpoint:

<YourSharePointSiteUrl>/_api/web/lists/getbytitle('<DocumentLibraryTitle>')/GetItems

Again you can use $select parameter to limit the information you want for each item returned.

For filtering, we use CAML Query:

The above CAML query translates to “Give me all the items that ID is greater than 0 and file name contains “test”.

Make a POST request to the API endpoint with above CAML query as body, you will get the response contains all the documents that meet the WHERE condition defined in the CAML query.

Note the FieldRef name in the CAML query is the internal name used in document library, not the display name. In order to found out what the internal name is, you can use the following API endpoint.

Field Meta Data API Endpoint

The field meta data API endpoint URL is:

<YourSharePointSiteUrl>/_api/web/lists/getbytitle('<DocumentLibraryTitle>')/Fields

Again, you can use the $select parameter to select the field you want to see.

?$select=Title,InternalName

Download document

Each item inside document library has a property called “EncodedAbsUrl”. This is the URL pointing to the file you can download.

Given the document ID, we can first call the Document Search API to get the list item. After that, we can use the EncodedAbsUrl property of the item to download the document. However, you may need to manually specify the MIME type based on the document file extension.

NTLM authentication in HttpClient

In asp.net core 3.1, we can use HttpClientHandler to pass the NTLM authentication. You can configure the HttpClient in the startup.cs file.

services.AddHttpClient("SharepointClient", c =>
{
    c.DefaultRequestHeaders.Add("Accept", "application/json;odata=verbose");
}).ConfigurePrimaryHttpMessageHandler(c =>
{
    var settings = Configuration.GetSection($"SharePointSettings").Get<SharePointSettings>();
    return new HttpClientHandler { Credentials = new NetworkCredential(settings.SPUserName, settings.SPPassword, settings.SPDomain) };
});

In the controller or service class, you can inject the IHttpClientFactory in the class constructor and create the named HttpClient:

public DocumentController(IHttpClientFactory factory)
{
    _httpClient = factory.CreateClient("SharepointClient");
}

You can find the source code in github here: https://github.com/jason-ge/SharePointCamlQuery.

In order to test it, you have to have a SharePoint environment. You need to change the appsettings.json file to put in the correct Windows credential that has access to your SharePoint site.

When you start the project in VisulaStudio 2019, it would launch Swagger page, you can test the API methods using Swagger UI.

Happy coding!