Bulk Metadata Management
BulkMetadataManager is a command-line application for SharePoint administrators. It runs on a Windows system in your own environment and works with SharePoint Online, SharePoint 2019 and SharePoint Server Subscription Edition.
One tool, three modes:
- Extract metadata – extract properties from emails, Office files, PDFs, images and videos into SharePoint columns, without a new version and without changing Modified / Modified by.
File types can be mapped to different content types in a single run. For example, msg and eml to Email, and binary Office files to a separate content type. Earlier versions needed a separate run per content type. - Detect duplicates – inventory of documents, store details in a local SQLite database, and find candidates (content hash, Message-ID, and related signals). Phases can run together or separately. Support for incremental runs.
- Bulk delete – read document URLs from a csv file and delete in per-site batch requests. Default: recycle bin (typically restorable for 93 days). Optional: permanent delete.
Typical metadata extraction examples include:
- The sent date of an email.
- Keywords embedded in a PDF.
- The last modified date of an Office document.
- The date taken of a photo.
- The GPS location from a drone recording.
These metadata values are automatically captured into SharePoint columns – without creating a new document version or altering the “Modified” or “Modified by” fields in SharePoint.
Supported File Types:
- Emails: msg and eml.
- Documents: docx, xlsx, pptx, vsdx, pdf, doc, xls and ppt.
- Images: jpg, png, dng, heic, heif, tiff, jfif and cr2.
- Videos: mp4, mov and wmv.
BulkMetadataManager also supports extracting unique words from email attachments (pdf, docx, xlsx, pptx, vsdx, xml, csv, txt, zip), enabling users and compliance officers to search for content within email attachments.
Non-functional characteristics:
- Built to handle large volumes of documents (100,000+).
- Optimized for high performance.
- Support for horizontal and vertical scaling.
- Secure.
Benefits: BulkMetadataManager empowers organizations to fully leverage the capabilities of the SharePoint platform, enhancing user productivity and adoption. Utilizing metadata in SharePoint columns allows for efficient filtering, sorting, grouping, and powerful searches. Additionally, it helps organizations meet compliance requirements and keeps storage consumption under control.
Click here for a 1-minute overview.
Select
Process
Apply
Product Features
See all features below
| Feature | BulkMetadataManager |
|---|---|
| Support for emails: msg and eml | ✓ |
| Support for documents: docx, xlsx, pptx, vsdx, pdf, doc, xls and ppt | ✓ |
| Support for images: jpg, png, dng, heic, heif, tiff, jfif and cr2 | ✓ |
| Support for videos: mp4, mov and wmv | ✓ |
| Duplicate detection: inventory + local SQLite + candidate matching | ✓ |
| Incremental duplicate runs: 'LastDuplicatesRunDate' | ✓ |
| Bulk delete: from csv file, batched per site | ✓ |
| Recycle bin or permanent delete: 'UseRecycleBin' | ✓ |
| Configurable: Select specific sites/libraries, use wildcards, exclusions, ... | ✓ |
| Granular Control: Select files using CAML queries | ✓ |
| Process different content types in a single run: map different file extensions to different content types | ✓ |
| Performance: Up to 5-10 documents per second | ✓ |
| Large List Support: Efficiently handles large lists | ✓ |
| Throttling Management: Caters for SharePoint throttling | ✓ |
| WhatIf Support: Provides WhatIf functionality | ✓ |
| Logging: Comprehensive logging capabilities | ✓ |
| Scalability: Designed to be scalable | ✓ |
| Extensibility: Extensible to support additional file types (e.g., zip, odt, ...) | ✓ |
| SharePoint Online: Entra ID app-only (certificate) | ✓ |
| SharePoint 2019 / SE: NTLM | ✓ |
| System Resources: low-end computer (e.g. laptop, desktop or small VM) | ✓ |
Leverage metadata and clean up content
Overview
BulkMetadataManager helps you take control of content already in SharePoint: enrich it with real metadata, find copies that waste storage and add compliance and audit risk, and remove what you no longer need.
Benefits: BulkMetadataManager enables organizations to fully leverage the SharePoint platform, enhancing productivity and user adoption.
- Enrich documents in SharePoint with metadata
- Search, filter and retain original ‘document’ dates – not the upload date
- Make email attachments searchable
- Find true duplicates (including emails) instead of relying on file name alone
- Delete at scale with logging; recycle by default so mistakes are recoverable
- No external SaaS: the application and database stay on a machine you manage
How It Works:
- Metadata – download (CAML) → extract → update columns.
- Duplicates – inventory → SQLite → compare → csv file/report.
- Delete – csv file with URLs → group by site → delete to the recycle bin or permanently.
Common Use Cases:
- Add metadata (e.g., subject, sent date, …) to emails stored in SharePoint to use SharePoint as an email archive.
- Extract order details from PDF files stored in SharePoint.
- Extract the ‘last modified’ dates from emails, PDF documents, (Binary) Office files, images and videos using the metadata embedded within the files (not the upload date), and automatically populate custom columns in SharePoint with these values.
- Make email attachments (pdf, docx, xlsx, pptx, vsdx, xml, csv, txt, zip) searchable.
- Extract GPS coordinates and ‘Date Taken’ from images/videos to create a digital library in SharePoint.
- Find duplicate invoices, presentations and emails across all site collections.
- After a merge or file-share migration, remove copies and keep one location.
- Recycle or permanently delete a reviewed list without clicking through the SharePoint UI.
BulkMetadataManager in-depth
Metadata extraction
The following figures show the document details before and after running the application.
| Metadata before | Metadata after |
|
|
Supported file types and properties
The following properties can be extracted from files stored in SharePoint:
| Emails (msg and eml) |
Office Files (docx, xlsx, pptx and vsdx) |
Binary Office Files (doc, xls and ppt) |
PDF Files (pdf) |
Images (jpg, png, dng, heic, heif, tiff, jfif and cr2) |
Videos (mp4, mov or wmv) |
| Subject From From address To To address CC CC address BCC BCC address Message ID Sent Date Received Date Conversation has Attachment Importance Sensitivity Categories Attachment count Attachment names Attachment contents (unique words) References ConversationGUID NormalizedSubject |
Title Tags Comments Status Categories Subject Company Manager Hyperlink Base Last Modified Created Last Printed Author Last Modified By Revision Legal Entity Project Language Total Editing Time Template Pages Words Characters (no spaces) Characters (with spaces) Paragraphs Lines Application DocSecurity AppVersion |
Title Comments Status Categories Subject Company Manager Author Create Time Last Printed Last Saved By Revision Number Total Editing Time Template Number of Pages Number of Words Number of Characters Paragraphs Lines Name of Creating Application Security AppVersion |
Title Author Comments Subject Keywords PDF Producer PDF Version Fast Web View Number of Pages Tagged PDF Modified Created Application Page Layout Metadata Date Format Document ID Instance ID Language |
DateTime GPS Coordinates GPS Latitude (DD) GPS Longitude (DD) GPS Altitude EXIF properties XMP properties IPTC properties ICC properties |
DateTime GPS Coordinates GPS Latitude (DD) GPS Longitude (DD) GPS Altitude EXIF properties XMP properties IPTC properties ICC properties |
Process different content types in a single run
BulkMetadataManager supports processing different content types mapped to different file extensions in a single operation. This is controlled via a property in the BulkMetadataManagerProperties.txt file. For example:
DocumentContentTypeName=Email(msg/eml),BinaryOfficeFile(doc/xls/ppt)
This maps msg and eml files to the Email content type, and binary Office files (doc, xls, ppt) to the BinaryOfficeFile content type.
Using the property DocumentContentTypeName without file extensions will result in applying the first matching content type for the items selected using a CAML query.
Recover original create and modify dates
BulkMetadataManager allows for using the original create and modify dates present within files as SharePoint Created and Modified dates.
Existing situation
After extraction the original create and modify dates are used as Created and Modified dates
Making Email Attachments Searchable
SharePoint Search does not index all email attachments. Our tests across different tenants indicate that the indexing rate is well below 50%, leading to significant blind spots in the search index. This limitation affects users and compliance officers, as emails may not appear in search results.
BulkMetadataManager addresses this issue by ensuring full visibility of SharePoint data, eliminating blind spots. In short, BulkMetadataManager is essential for complying with privacy and other regulations.
Duplicate detection
Using BulkMetadataManager with the ‘Mode=Duplicates’ builds an inventory of documents (path, size, SHA256 hashes and email identifiers such as Message-ID) in a SQLite database created on the same system as the BulkMetadataManager application. A second phase compares candidates using a sophisticated algorithm:
- For documents unaffected by property demotion (pdf, jpg, mp4, …) the SHA256 hash is used. Special logic is applied to large files to minimize download the full files.
- BulkMetadataManager automatically handles documents affected by property demotion like docx, pptx, …
- For emails the unique value of the MessageID is used.
Use ‘Mode=Duplicates’ to run the full process or run inventory and comparison separately. Set ‘LastDuplicatesRunDate’ to process only documents changed since the last run.
By default the PreservationHoldLibrary is excluded. This library can be included by using setting the property ‘IncludePreservationHoldLibrary=true’.
The output is a list in csv format for further analysis. The csv file can be used for the Delete mode.
Bulk deletion
‘Mode=Delete’ reads URLs from a csv file. Requests are grouped by site and sent as SharePoint batch operations.
Default: items go to the recycle bin. With standard first- and second-stage settings they remain restorable for 93 days.
Permanent delete: ‘UseRecycleBin=false’. Log lines are marked “Permanently deleted”.
Retention labels, records and eDiscovery holds still block deletion; those items are logged as failed, not removed.
Delete requires write permission on the target sites (for example ‘SharePoint: Sites.ReadWrite.All’, or ‘SharePoint: Sites.Selected’ with the Write role). Read-only app permissions are enough for inventory and detection.
Logging
The application generates multiple log files to provide comprehensive insights:
- Summary Log File: Contains high-level details.
- Detailed Log File: Includes specific details for individual documents.
- WhatIf Log File: Provides details of the extracted properties when the WhatIf property is set to true.
Performance
The application is optimized for high performance through the use of parallel processes for downloading documents and setting metadata, caching, and other optimization techniques. Under optimal conditions, the application can process 10 documents per second. It is designed to handle very large volumes of documents (100,000+). Additionally, the application architecture supports scaling out by utilizing separate computer systems.
BulkMetadataManager can manage throttling by SharePoint Online and will automatically resume operations after the retry-after period has expired. The application also supports large lists that exceed the list view threshold (5,000 items).
Scalability
BulkMetadataManager supports both vertical and horizontal scaling. Vertical scaling allows for increased processing throughput without the need to deploy additional infrastructure. This setup ensures scalability and isolation between instances while maintaining efficient metadata processing.
Horizontal scaling uses different systems to increase the processing throughput.
Secure
BulkMetadataManager uses Entra ID app-only authentication with a certificate to securely access SharePoint Online.
It gives you fine-grained control, allowing access to be granted to all sites or only to selected sites. The application runs entirely within your own IT environment, with no dependency on external cloud services or third-party systems. It operates natively without triggering Power Automate flows.
After processing, the original Modified date and Modified By information are fully retained, and no additional document version is created that would count toward your storage quota.
Delete and inventory use the same Entra ID certificate as metadata extraction. Grant ‘write’ only if you use Delete mode.
WhatIf
The WhatIf option enables assessment of the application and its configuration without uploading metadata to SharePoint. Extracted document properties are saved to a separate csv file for further analysis.
Deployment
Deployment involves the following steps:
- Install Application: Install the BulkMetadataManager application (exe) on a separate computer (e.g., small VM, laptop, desktop, …).
- Authentication/Authorization: Configure Entra ID app-only authentication using a certificate (detailed instructions available)
- Add/Deploy Content Type: Add or deploy the Content Type to the SharePoint libraries (if not already present).
- Configure Metadata Mapping: Set up metadata mapping using a static properties file or tenant properties.
- Configure ini File: Adjust the ini file for your environment to allow granular control over which sites and/or libraries are processed.
- Update CAML File: Modify the CAML file to select the documents of interest.
- Run Tool: Execute the exe tool via the command line or via scheduled task
- For duplicates: the SQLite database is created automatically on the same machine as the exe.
- For delete: configure the ini with the csv file containing the URLs; optionally set ‘UseRecycleBin=false’
BulkMetadataManager can extract metadata from documents regardless of how they were uploaded to SharePoint — whether through the web interface, OneDrive for Business, mobile apps, Power Automate, or external applications.
Granular Control
The use of a CAML query allows for granular control over which documents are processed. BulkMetadataManager will only process documents selected using the CAML query. The CAML queries intelligently identify and skip already-processed items, preventing unnecessary re-processing.
Supported Environments
BulkMetadataManager supports SharePoint Online, SharePoint Server SE (via NTLM authentication) and SharePoint 2019 (via NTLM authentication).
Enhancements
The following enhancements will be implemented in future releases:
- Support for importing metadata from CSV files (to complement the extracted document properties).
- Support for detecting Privacy & Confidentiality terms (e.g., social security number) in emails and email attachments.
- Enable extraction of file creation and modification dates from SharePoint documents, and update the corresponding ‘Created’ and ‘Modified’ columns in SharePoint accordingly.
- Support for adding retention labels to documents.
- Support for emails with binary Office attachments (doc, xls, ppt).
- Review UI for keep/delete per duplicate group
Target Audience
BulkMetadataManager is intended for use by SharePoint administrators and site collection administrators. It is not designed for end-users.
For Further Information
Check out the following resources for more details:
In short, BulkMetadataManager extracts metadata, finds duplicate documents, and deletes at scale – inside your network, with the audit trail and storage behavior you configure.