Migration of Work items with history on now removed area paths

[jeffpriz]

I have a migration of a set of work items running. I've mapped area paths from the source to the target.. including some area paths that are found in the history of the work items in question... however, I am running in to an issue where it wants to re-create those old area paths in the Source project. I don't know why this is the case.

I did not have rights to create paths in the source project, only the target project. The Area Paths in question have value maps in my config to map to the appropriate target area paths. The migration would fail with "User Permissions" creating nodes.

I have mapping for that path in question that it says is not present in the Target

Now with rights in the Source team project, I can run past that issue. The process then proceeds to re-create area paths in the source?

Not sure what's going on here, or if I've messed something up? or why these paths need to be recreated. Any thoughts/Help are appreciated!

[MrHinsh]

No changes are ever made to the Source project. We only ever read from the Source...

HOWEVER! The old TFS Object Model requires write access as it doe no support granular permissions and thus will crap out if it does not have the permission needed for a possible action that it might take.

We have a list of permissions that might work on https://github.com/nkdAgility/azure-devops-migration-tools?tab=readme-ov-file#advanced-tools! If something is missing I'd happily add it, or you can submit a PR with the change.

[jeffpriz]

@MrHinsh I found that the Area Paths in question were in fact re-created in the source project, which was unexpected, and which is why I posted this question here. I am not sure why/how that is done. There were area paths that were cleaned up in the last few months, and removed.. no work items were left in those area paths and the area paths were removed. When migrating Work Items with historical information in the now-deleted area paths the migrations failed (with the permissions error above)... once the rights were granted in the Source project to re-create these removed area paths the migration was able to proceed.. and the area paths in the source project were re-created.

I am open to the idea that i mis-configured something that cause that behavior, I just am a bit confused as to what configuration causes that behavior.

[jeffpriz]

@MrHinsh Here is an example screen shot of the Area Path tree in the SOURCE project after executing the migration from here.. Note the "All Stories Not Sprint Ready" tree paths.. they did not exist in this tree just prior to the execution of the Migration.. You'll also notice that these are the paths that were flagged by the error in my original post. I will try to clean-up and re-execute the migration to capture the messages in the output where it said it created these nodes.

[MrHinsh]

I checked through the code and nowhere are changes pushed to the source that I can find..

[MrHinsh]

I just fully checked throug TfsNodeStructure and here are all the calls to the ICommonStructureService4 for the Source:

// Redacted code...

protected override void EntryForProcessorType(IProcessor processor)
{
    if (processor is null)
    {
        IMigrationEngine engine = Services.GetRequiredService<IMigrationEngine>();
        if (_sourceCommonStructureService is null)
        {
            _sourceCommonStructureService = (ICommonStructureService4)engine.Source.GetService<ICommonStructureService4>();
            _sourceProjectInfo = _sourceCommonStructureService.GetProjectFromName(engine.Source.Config.AsTeamProjectConfig().Project);
            _sourceRootNodes = _sourceCommonStructureService.ListStructures(_sourceProjectInfo.Uri);
            _sourceLanguageMaps = engine.Source.Config.AsTeamProjectConfig().LanguageMaps;
            _sourceProjectName = engine.Source.Config.AsTeamProjectConfig().Project;
        }
    }
    else
    {
        if (_sourceCommonStructureService is null)
        {
            var source = (TfsWorkItemEndpoint)processor.Source;
            _sourceCommonStructureService = (ICommonStructureService4)source.TfsCollection.GetService<ICommonStructureService>();
            _sourceProjectInfo = _sourceCommonStructureService.GetProjectFromName(source.Project);
            _sourceRootNodes = _sourceCommonStructureService.ListStructures(_sourceProjectInfo.Uri);
            _sourceLanguageMaps = source.Options.LanguageMaps;
            _sourceProjectName = source.Project;
        }
    }
}

// Redacted code...

private void ProcessCommonStructure(string treeTypeSource, string localizedTreeTypeName, string projectTarget, TfsNodeStructureType nodeStructureType)
{
    var startPath = ("\\" + this._sourceProjectName + "\\" + treeTypeSource).ToLower();
    var nodes = _sourceRootNodes.Where(n => n.Path.ToLower().StartsWith(startPath));
    if (nodes.Count() > 1)
    {
        Exception ex = new Exception(string.Format("Unable to load Common Structure for Source because more than one node path matches \"{0}\". {1}", treeTypeSource, JsonConvert.SerializeObject(nodes.Select(x => x.Path))));
        throw ex;
    }
    NodeInfo sourceNode = nodes.FirstOrDefault();
    if (sourceNode == null)
    {
        Exception ex = new Exception(string.Format("Unable to load Common Structure for Source. This is usually due to different language versions. Validate that '{0}' is the correct name in your version. ", treeTypeSource));
        throw ex;
    }
    XmlElement sourceTree = _sourceCommonStructureService.GetNodesXml(new string[] { sourceNode.Uri }, true);
    // Redacted code...
}

private void PopulateIterationDatesFronSource(NodeStructureItem missingItem)
{
    if (missingItem.nodeType == "Iteration")
    {
        try
        {
            var sourceNode = _sourceCommonStructureService.GetNodeFromPath(missingItem.sourceSystemPath);
            missingItem.startDate = sourceNode.StartDate;
            missingItem.finishDate = sourceNode.FinishDate;
            missingItem.sourcePathExists = true;
        }
        catch (Exception)
        {
            missingItem.startDate = null;
            missingItem.finishDate = null;
            missingItem.sourcePathExists = false;
        }
    }
}

public string FixAreaPathAndIterationPathForTargetQuery(string sourceWIQLQuery, string sourceProject, string targetProject, ILogger? contextLog)
{
    string targetWIQLQuery = sourceWIQLQuery;

    if (string.IsNullOrWhiteSpace(targetWIQLQuery))
    {
        return targetWIQLQuery;
    }

    var matches = Regex.Matches(targetWIQLQuery, RegexPatternForAreaAndIterationPathsFix);

    if (string.IsNullOrWhiteSpace(sourceProject)
        || string.IsNullOrWhiteSpace(targetProject)
        || sourceProject == targetProject)
    {
        return targetWIQLQuery;
    }

    foreach (Match match in matches)
    {
        var value = match.Groups["value"].Value;
        if (string.IsNullOrWhiteSpace(value) || !value.StartsWith(sourceProject))
            continue;

        var fieldType = match.Groups["key"].Value;
        TfsNodeStructureType structureType;
        switch (fieldType)
        {
            case "System.AreaPath":
                structureType = TfsNodeStructureType.Area;
                break;
            case "System.IterationPath":
                structureType = TfsNodeStructureType.Iteration;
                break;
            default:
                throw new InvalidOperationException($"Field type {fieldType} is not supported for query remapping.");
        }

        var remappedPath = GetNewNodeName(value, structureType);
        targetWIQLQuery = targetWIQLQuery.Replace(value, remappedPath);
    }

    contextLog?.Information("[FilterWorkItemsThatAlreadyExistInTarget] is enabled. Source project {sourceProject} is replaced with target project {targetProject} on the WIQLQuery which resulted into this target WIQLQueryBit \"{targetWIQLQueryBit}\" .", sourceProject, targetProject, targetWIQLQuery);

    return targetWIQLQuery;
}

// Redacted code...

At no time are there any writes to the source project!

I feel like this can only happen if the target project and the source project are the same project!

[jeffpriz]

@MrHinsh thanks for the thorough check. The Source Project and the Target project are in the same organization but are different projects. I will double-triple verify that I am seeing what I think I am seeing and reply here later with what I know.

[jeffpriz]

for reference here is my source/target definition.. the Organizations are the same org. The Projects are different