Bulk metadata management
BulkMetadataManager is an application that extracts properties from existing SharePoint documents (e.g., sent date of an email, keywords in a pdf file, “Last Modified” date from an Office file, GPS location from a drone recording, …) and captures the extracted values into SharePoint columns.
The following file types are supported:
– Emails (msg and eml)
– Documents (docx, xlsx, pptx, vsdx and pdf)
– Images (jpg, png, dng, heic, heif, tiff, jfif and cr2)
– Videos (mp4, mov and wmv)
BulkMetadataManager also supports extracting unique words from email attachments (pdf, docx, xlsx, pptx, vsdx, xml, csv, txt and zip) to allow users / compliance officers to search for content in email attachments.
The application is build for handling large number of documents (100000’s+) and is optimized for performance.
BulkMetadataManager enables organizations to make optimum use of the capabilities of the SharePoint platform and improve user productivity and adoption. Using metadata in SharePoint columns allows for filtering, sorting, grouping and powerful searches. Lastly, it helps organizations meet compliance requirements (e.g., destroy content 7 years after the last modification).
Select documents using CAML
Extract properties from documents
Upload metadata to SharePoint
Product Features
See all features below
| Functionality | BulkMetadataManager |
|---|---|
| Extract metadata from emails (msg and eml) stored in SharePoint | ✓ |
| Select emails using CAML | ✓ |
| Performance | Typically 10 documents per second |
| Support for large lists | ✓ |
| Caters for SharePoint throttling | ✓ |
| WhatIf support | ✓ |
| Logging | ✓ |
| Scalable | ✓ |
| Extensible | ✓ (e.g., pdf, jpg, docx, ...) |
Leverage metadata
Overview
BulkMetadata extracts properties present within SharePoint documents (e.g., sent date, subject, … from emails, keywords from PDF files, last modified date from Office files), and captures the extracted values into SharePoint columns. The application runs on a separate computer system and does not require any user interactions.
The application enables organizations to make optimum use of the SharePoint platform and improve productivity and user adoption. Using metadata in SharePoint columns allows for filtering, sorting, grouping and powerful searches. By extracting the created / modified dates from the actual documents (docx, pdf, msg, jpg, …) and not the import date, organizations can ensure data is retained/destroyed in line with legal requirements.
Basically, the application:
– downloads documents based on a CAML query
– extracts the document properties
– uploads the metadata to SharePoint columns.
Common use cases:
– add metadata (e.g. sent date) to emails stored in SharePoint
– extract order details from PDF files stored in SharePoint
– extract last modified dates from emails, pdf files and Office files in SharePoint using the properties stored within the files (and not the date when the document was uploaded to SharePoint)
– make email attachments (pdf, docx, xlsx, pptx, vsdx, xml, csv, txt and zip) searchable
– extract GPS coordinates and ‘DateTimeTaken’ from images/videos to create a digital library in SharePoint
BulkMetadataManager in-depth
The following figures show the document details before and after running the application.
Metadata before:
Metadata after:
Supported file types and properties
The following properties can be extracted from documents stored in SharePoint:
Recover original create and modify dates
BulkMetadataManager allows for using the original create and modify dates present within files as SharePoint Created and Modified dates.
Existing situation
After extraction the original create and modify dates are used as Created and Modified dates
Making email attachments searchable
SharePoint Search does not index all email attachments. Our tests in different tenants indicate this is well below 50%. This results in blind spots in the search index. This affects users (and compliance officers) because search will not include the emails in the search results list. BulkMetadataManager allows organizations to have full visibility of their SharePoint data without having many blind spots. In short, BulkMetadataManager is a critical component to comply with privacy and other regulations.
Logging
The application generates multiple log files:
– a summary log file with high level details
– a detailed log file with details for the individual documents
– a whatif log file with the details of the extracted properties (if the WhatIf property is set to true)
Performance
The application is optimized for performance by using parallel processes for downloading documents and setting metadata, caching, and other optimization techniques. Under optimized conditions the application can process 10 documents per second. The application is build to handle very large number of documents (100000’s and more). In addition, the application architecture allows for scaling out by using separate computer systems.
BulkMetadataManager can handle throttling by SharePoint Online and will automatically resume after the retry-after period has expired. The application also supports large lists exceeding the list view threshold (5000 items).
WhatIf
The WhatIf option allows for assessing the application and the configuration without uploading the metadata to SharePoint. The extracted document properties are saved to a separate csv file for further analysis.
Deployment
This comprises of the following steps:
– install the BulkMetadataManager application (exe) on a separate computer (e.g., smallest Azure VM server, laptop, …)
– add/deploy Content Type to the SharePoint librarie(s) (if not already present)
– configure the metadata mapping using a static properties file (or using tenant properties)
– configure the ini file for your environment (this allows for granular control which sites and/or libraries are processed)
– update the CAML file to select the documents of interest
– run the tool via the cmd line
BulkMetadataManager can also be scheduled via the standard Windows Task Scheduler. This allows for extracting metadata from documents independent of the way they have been added to SharePoint (via web interface, OneDrive for Business, mobile applications, Power Automate, external applications, …).
Granular control
The use of a CAML query allows for granular control which documents are to be processed. BulkMetadataManager will only process documents selected using the CAML query and where the content type is present in the document’s library.
Supported environments
BulkMetadataManager supports SharePoint Online. Contact us if you have requirements for on-premise SharePoint versions.
Enhancements
The following enhancements will be implemented in future releases:
– support for importing metadata from csv files (to complement the extracted document properties)
– support for SharePoint 2019 and SharePoint Server SE
– support for labelling documents (retention/sensitivity)
– support for binary Office files (doc, xls and ppt)
The target audience for running BulkMetadataManager is SharePoint administrators and site collection administrators. It is not intended for use by end-users.
Additional information
For further information check out the following sources:
In short: BulkMetadataManager offers unique functionality to manage the metadata of your SharePoint documents. It improves user productivity and enhances compliance with privacy, archiving and other regulations.