UnexpectedErrrorException - Contact your DevOps Server Administrator

[jeffpriz]

Running some dry run migration activities in preparation of an upcoming migration of a project's work items to a new project in AzDO. Getting an Odd exception that I don't think I've seen before, and it doesn't have a lot of info.. and it feels fairly random (in that if I clean things up from the run, and restart it, it will fail on a different work item the next time i run it, but succeed on the one it failed on the last time). Also this is the 2nd or 3rd dry run of migration of this project I've done, and not had any issues.. on this run the one change I put into the config was to update some field mappings for some Iteration Paths that had changed after the last dry run. None of the work items this has failed on have been in any of the effected new iterations. I wonder if this is just a Microsoft Azure DevOps issue/outage, or if there is something I can do about it.

{"Timestamp":"2025-02-05T17:40:39.9512941-06:00","Level":"Error","MessageTemplate":"Microsoft.TeamFoundation.WorkItemTracking.Client.UnexpectedErrorException: There is a problem on the server. Contact your Azure DevOps Server administrator. ---> System.Web.Services.Protocols.SoapException: bd7f9640-d98c-43b9-b3b0-d62e03911701\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.RetryHandler.HandleSoapException(SoapException se)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)\r\n --- End of inner exception stack trace ---\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SaveWorkItems(WorkItem[] workitems, Boolean bulk, SaveFlags saveFlags)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.Save(SaveFlags saveFlags)\r\n at MigrationTools.TfsExtensions.SaveToAzureDevOps(WorkItemData context) in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\TfsExtensions.cs:line 83\r\n at MigrationTools.Processors.TfsWorkItemMigrationProcessor.ReplayRevisions(List`1 revisionsToMigrate, WorkItemData sourceWorkItem, WorkItemData targetWorkItem) in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 817","TraceId":"30f7d1ff827dd74cebd6444b88d78964","SpanId":"d22177bda8b6958f","Properties":{"SourceContext":"MigrationTools.Processors.TfsWorkItemMigrationProcessor","versionString":"16.0.9","targetWorkItemId":null,"sourceRevisionInt":43,"sourceWorkItemId":136919,"totalWorkItems":4694,"currentWorkItem":369,"sourceWorkItemTypeName":"Feature","ProcessId":31128}}

{"Timestamp":"2025-02-05T17:40:40.0296486-06:00","Level":"Error","MessageTemplate":"Microsoft.TeamFoundation.WorkItemTracking.Client.UnexpectedErrorException: There is a problem on the server. Contact your Azure DevOps Server administrator. ---> System.Web.Services.Protocols.SoapException: bd7f9640-d98c-43b9-b3b0-d62e03911701\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.RetryHandler.HandleSoapException(SoapException se)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)\r\n --- End of inner exception stack trace ---\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SaveWorkItems(WorkItem[] workitems, Boolean bulk, SaveFlags saveFlags)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.Save(SaveFlags saveFlags)\r\n at MigrationTools.TfsExtensions.SaveToAzureDevOps(WorkItemData context) in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\TfsExtensions.cs:line 83\r\n at MigrationTools.Processors.TfsWorkItemMigrationProcessor.d__31.MoveNext() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 571","TraceId":"30f7d1ff827dd74cebd6444b88d78964","SpanId":"d5e1ada9d4e8d33d","Exception":"Microsoft.TeamFoundation.WorkItemTracking.Client.UnexpectedErrorException: There is a problem on the server. Contact your Azure DevOps Server administrator. ---> System.Web.Services.Protocols.SoapException: bd7f9640-d98c-43b9-b3b0-d62e03911701\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.RetryHandler.HandleSoapException(SoapException se)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)\r\n --- End of inner exception stack trace ---\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SaveWorkItems(WorkItem[] workitems, Boolean bulk, SaveFlags saveFlags)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.Save(SaveFlags saveFlags)\r\n at MigrationTools.TfsExtensions.SaveToAzureDevOps(WorkItemData context) in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\TfsExtensions.cs:line 83\r\n at MigrationTools.Processors.TfsWorkItemMigrationProcessor.d__31.MoveNext() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 571","Properties":{"SourceContext":"MigrationTools.Processors.TfsWorkItemMigrationProcessor","versionString":"16.0.9","targetWorkItemId":null,"sourceRevisionInt":43,"sourceWorkItemId":136919,"totalWorkItems":4694,"currentWorkItem":369,"sourceWorkItemTypeName":"Feature","ProcessId":31128}}

{"Timestamp":"2025-02-05T17:40:40.1627723-06:00","Level":"Error","MessageTemplate":"Could not save migrated work item {WorkItemId}, an exception occurred.","TraceId":"30f7d1ff827dd74cebd6444b88d78964","SpanId":"d5e1ada9d4e8d33d","Exception":"System.AggregateException: One or more errors occurred. ---> Microsoft.TeamFoundation.WorkItemTracking.Client.UnexpectedErrorException: There is a problem on the server. Contact your Azure DevOps Server administrator. ---> System.Web.Services.Protocols.SoapException: bd7f9640-d98c-43b9-b3b0-d62e03911701\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.RetryHandler.HandleSoapException(SoapException se)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)\r\n --- End of inner exception stack trace ---\r\n at MigrationTools.Processors.TfsWorkItemMigrationProcessor.d__31.MoveNext() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 621\r\n --- End of inner exception stack trace ---\r\n at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)\r\n at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.Task.Wait()\r\n at MigrationTools.Processors.TfsWorkItemMigrationProcessor.InternalExecute() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 178\r\n---> (Inner Exception #0) Microsoft.TeamFoundation.WorkItemTracking.Client.UnexpectedErrorException: There is a problem on the server. Contact your Azure DevOps Server administrator. ---> System.Web.Services.Protocols.SoapException: bd7f9640-d98c-43b9-b3b0-d62e03911701\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.RetryHandler.HandleSoapException(SoapException se)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)\r\n --- End of inner exception stack trace ---\r\n at MigrationTools.Processors.TfsWorkItemMigrationProcessor.d__31.MoveNext() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 621<---\r\n","Properties":{"WorkItemId":136919,"FieldValue":"Migration Sandbox","FieldName":"Iteration Path","FieldReferenceName":"System.IterationPath","RevisionNumber":36,"RevisionsToMigrateCount":43,"TargetWorkItemId":"454244","ReplayRevisions":true,"sourceWorkItemRev":43,"SourceWorkItemId":136919,"SourceContext":"MigrationTools.Processors.TfsWorkItemMigrationProcessor","versionString":"16.0.9","targetWorkItemId":null,"sourceRevisionInt":43,"sourceWorkItemId":136919,"totalWorkItems":4694,"currentWorkItem":369,"sourceWorkItemTypeName":"Feature","ProcessId":31128}}

{"Timestamp":"2025-02-05T17:40:40.7009920-06:00","Level":"Fatal","MessageTemplate":"Error while running {MigrationContextname}","TraceId":"30f7d1ff827dd74cebd6444b88d78964","SpanId":"d5e1ada9d4e8d33d","Exception":"System.AggregateException: One or more errors occurred. ---> Microsoft.TeamFoundation.WorkItemTracking.Client.UnexpectedErrorException: There is a problem on the server. Contact your Azure DevOps Server administrator. ---> System.Web.Services.Protocols.SoapException: bd7f9640-d98c-43b9-b3b0-d62e03911701\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.RetryHandler.HandleSoapException(SoapException se)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)\r\n --- End of inner exception stack trace ---\r\n at MigrationTools.Processors.TfsWorkItemMigrationProcessor.d__31.MoveNext() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 621\r\n --- End of inner exception stack trace ---\r\n at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)\r\n at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.Task.Wait()\r\n at MigrationTools.Processors.TfsWorkItemMigrationProcessor.InternalExecute() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 204\r\n at MigrationTools.Processors.Infrastructure.Processor.Execute() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools\Processors\Infrastructure\Processor.cs:line 99\r\n---> (Inner Exception #0) Microsoft.TeamFoundation.WorkItemTracking.Client.UnexpectedErrorException: There is a problem on the server. Contact your Azure DevOps Server administrator. ---> System.Web.Services.Protocols.SoapException: bd7f9640-d98c-43b9-b3b0-d62e03911701\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.RetryHandler.HandleSoapException(SoapException se)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)\r\n at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)\r\n --- End of inner exception stack trace ---\r\n at MigrationTools.Processors.TfsWorkItemMigrationProcessor.d__31.MoveNext() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 621<---\r\n","Properties":{"MigrationContextname":"TfsWorkItemMigrationProcessor","SourceContext":"MigrationTools.Processors.TfsWorkItemMigrationProcessor","versionString":"16.0.9","ProcessId":31128}}
{"Timestamp":"2025-02-05T17:40:40.9144542-06:00","Level":"Error","MessageTemplate":"{Context} The Processor {ProcessorName} entered the failed state...stopping run","TraceId":"30f7d1ff827dd74cebd6444b88d78964","SpanId":"38ccda48a5789bed","Properties":{"Context":"TfsWorkItemMigrationProcessor","ProcessorName":"MigrationEngine","SourceContext":"MigrationTools.MigrationEngine","versionString":"16.0.9","ProcessId":31128}}

[MrHinsh]

Looks like your server is crapping out. Is it local?

[jeffpriz]

Nope, Azure DevOps Services in the cloud... Not a AzDO Server installation.. the full on AzDO Services... I've been running migrations for this client for months without troubles.. but the last couple days have been trouble

[jeffpriz]

I've checked the health status here, and have seen no issues :(
https://status.dev.azure.com/

Also, I've had failures at different points.. a couple failures yesterday afternoon happened within a half hour of the start of the run migrating items.. the migration only succeeded in migrating a couple dozen items... This last run I started in the evening to (hopefully) run overnight ran for close to two hours before failure... migrated several hundred items before failure

[jeffpriz]

I was able to complete this work item migration without further incident.
I will hope there was just something down on the Microsoft side the other day when I was experiencing this issue. Thanks to Hinsh for the reply. I was unsure if there was anything I could to do correct the trouble i was suddenly experiencing, but I think it was just a server-side issue at this point